Method for operating speech recognition service, electronic device and system supporting the same

ABSTRACT

An electronic device is provided. The electronic device includes a communication module, a microphone receiving a voice input according to user speech, a memory storing information about an operation of the speech recognition service, a display, and a processor electrically connected with the communication module, the microphone, the memory, and the display. The processor is configured to calculate a specified numerical value associated with the operation of the speech recognition service, to transmit information about the numerical value to a first external device processing the voice input, and to transmit a request for a function, which corresponds to the calculated numerical value, of at least one function associated with the speech recognition service stepwisely provided from the first external device depending on a numerical value, to the first external device to refine a function of the speech recognition service supported by the electronic device.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. §119(a) of a Korean patent application number 10-2017-0039589, filed onMar. 28, 2017, in the Korean Intellectual Property Office, and of aKorean patent application number 10-2017-0071017, filed on Jun. 7, 2017,in the Korean Intellectual Property Office, the disclosure of each ofwhich is incorporated by reference herein its entirety.

TECHNICAL FIELD

The disclosure relates to a function refinement technology of a speechrecognition service.

BACKGROUND

For the purpose of interaction with a user, recent electronic deviceshave proposed various input methods. For example, an electronic devicemay support a voice input method that receives voice data according touser speech, based on the execution of a specified application.Furthermore, the electronic device may recognize the received voice datato derive a user's speech intent and may support a speech recognitionservice performing an operation corresponding to the derived speechintent.

The speech recognition service may be implemented based on an artificialintelligence system using a machine learning algorithm The artificialintelligence system refers to a system that trains and determines byitself and improves a recognition rate as it is used, as a computersystem in which human intelligence is implemented. The artificialintelligence technology may include a machine learning (e.g., deeplearning) technology using an algorithm that classifies/learns thecharacteristics of input data by itself or elemental technologies (e.g.,a language understanding technology that recognizes thelanguage/character of a human, a reasoning/predicting technology thatdetermines information to logically infer and predict the determinedinformation, or the like) that simulate the functions such as therecognition of the human brain, the judgment of the human brain, and thelike by using the machine learning algorithm

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

In the operation of a speech recognition service, pieces of informationassociated with a user and an electronic device may be used as animportant resource for clearly recognizing voice data or for derivinguser utterance intent. As such, for the purpose of operating thereliable speech recognition service, the pieces of information need tobe collected appropriately and continuously. However, in the initialoperation of the speech recognition service, since the amount ofinformation about a user or an electronic device is very small, thespeech recognition rate may be low or an error may occur frequently.This may reduce the operating efficiency or reliability of the speechrecognition service.

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea speech recognition service operating method that updates a speechrecognition service through interaction with a user.

Another aspect of the disclosure is to provide an apparatus and methodfor refined speech recognition service function, an electronic deviceand system supporting the same.

In accordance with an aspect of the disclosure, a method for anelectronic device supporting a speech apparatus is provided. The methodincludes a communication module communicating with at least one externaldevice, a microphone receiving a voice input according to user speech, amemory storing information about an operation of the speech recognitionservice, a display outputting a screen associated with the operation ofthe speech recognition service, and a processor electrically connectedwith the communication module, the microphone, the memory, and thedisplay.

In accordance with another aspect of the disclosure, the processor isconfigured to calculate a specified numerical value associated with theoperation of the speech recognition service, to transmit informationabout the numerical value to a first external device processing thevoice input, and to transmit a request for a function, which correspondsto the calculated numerical value, of at least one function associatedwith the speech recognition service stepwisely provided from the firstexternal device depending on a numerical value, to the first externaldevice to refine a function of the speech recognition service supportedby the electronic device.

According to various embodiments, the reliable service may be providedto a user by refining the function of the speech recognition servicebased on the experience point (or growth points) of the artificialintelligence assistant supporting the operation of a speech recognitionservice.

According to various embodiments, the experience scenario organicallyassociated with a user may be provided to an artificial intelligenceassistant, and thus various infotainment environments may be providedupon operating a speech recognition service.

Besides, a variety of effects directly or indirectly understood throughthis disclosure may be provided.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1A is a block diagram illustrating an integrated intelligentsystem, according to an embodiment of the disclosure;

FIG. 1B is a block diagram illustrating a user terminal of an integratedintelligent system, according to an embodiment of the disclosure;

FIG. 1C is a view for describing how an intelligence app of a userterminal is executed, according to an embodiment of the disclosure;

FIG. 1D is a block diagram illustrating an intelligence server of anintegrated intelligent system, according to an embodiment of thedisclosure;

FIG. 1E is a view illustrating a method in which a natural languageunderstanding (NLU) module generates a path rule, according to anembodiment of the disclosure;

FIG. 2A is a view illustrating an interface associated with a speechrecognition service, according to an embodiment of the disclosure;

FIG. 2B is a view illustrating an interface associated with a speechrecognition service, according to an embodiment of the disclosure;

FIG. 2C is a view illustrating an interface associated with a speechrecognition service, according to an embodiment of the disclosure;

FIG. 3A is a view illustrating a speech recognition service operatingmethod of a user terminal, according to an embodiment of the disclosure;

FIG. 3B is a view illustrating a speech recognition service operatingmethod of an intelligence server, according to an embodiment of thedisclosure;

FIG. 4A is a flowchart illustrating a first embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an ASR module, according to an embodiment of thedisclosure;

FIG. 4B is a flowchart illustrating a second embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an ASR module, according to an embodiment of thedisclosure;

FIG. 4C is a flowchart illustrating a third embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an ASR module, according to an embodiment of thedisclosure;

FIG. 4D is a view illustrating an embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure;

FIG. 4E is a view illustrating another embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure;

FIG. 5A is a flowchart illustrating a fourth embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an NLU module, according to an embodiment of thedisclosure;

FIG. 5B is a view illustrating an embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure;

FIG. 5C is a flowchart illustrating a fifth embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an NLU module, according to an embodiment of thedisclosure;

FIG. 5D is a flowchart illustrating a sixth embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an NLU module, according to an embodiment of thedisclosure;

FIG. 5E is a view illustrating an embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure;

FIG. 6A is a flowchart illustrating a seventh embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of a personal information server, according to anembodiment of the disclosure;

FIG. 6B is a view illustrating an embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure;

FIG. 6C is a view illustrating another embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure;

FIG. 7A is a view illustrating an eighth embodiment to calculate anexperience point of an artificial intelligence assistant based onexecution of user activity, according to an embodiment of thedisclosure;

FIG. 7B is a view illustrating a ninth embodiment to calculate anexperience point of an artificial intelligence assistant based onexecution of user activity, according to an embodiment of thedisclosure;

FIG. 7C is a view illustrating a tenth embodiment to calculate anexperience point of an artificial intelligence assistant based onexecution of user activity, according to an embodiment of thedisclosure;

FIG. 7D is a view illustrating an eleventh embodiment to calculate anexperience point of an artificial intelligence assistant based onexecution of user activity, according to an embodiment of thedisclosure;

FIG. 8A is a view illustrating a first embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 8B is a view illustrating a second embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 8C is a view illustrating a third embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 8D is a view illustrating a fourth embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 8E is a view illustrating a fifth embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 8F is a view illustrating a sixth embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 8G is a view illustrating a seventh embodiment to use an experiencepoint of an artificial intelligence assistant, according to anembodiment of the disclosure;

FIG. 9A is a block diagram illustrating an architecture associated withsome elements in an integrated intelligent system, according to anembodiment of the disclosure;

FIG. 9B is a view illustrating a first process between some elements inan integrated intelligent system associated with an architecture,according to an embodiment of the disclosure;

FIG. 9C is a view illustrating a second process between some elements inan integrated intelligent system associated with an architecture,according to an embodiment of the disclosure;

FIG. 9D is a view illustrating a third process of some elementsexecution associated with an architecture, according to an embodiment ofthe disclosure;

FIG. 9E is a view illustrating a fourth process of some elementsexecution associated with an architecture, according to an embodiment ofthe disclosure;

FIG. 9F is a view illustrating an output example of various interfacesof a user terminal associated with promotion participation, according toan embodiment of the disclosure;

FIG. 9G is a view illustrating a fifth process between some elements inan integrated intelligent system associated with an architecture,according to an embodiment of the disclosure;

FIG. 9H is a view illustrating an output example of various interfacesof a user terminal associated with first user variation suggestion,according to an embodiment of the disclosure;

FIG. 9I is a view illustrating an output example of various interfacesof a user terminal associated with second user variation suggestion,according to an embodiment of the disclosure;

FIG. 9J is a view illustrating an output example of various interfacesof a user terminal associated with user variation suggestion adoption,according to an embodiment of the disclosure;

FIG. 9K is a view illustrating an interface output example of a userterminal associated with experience point limit excess of an artificialintelligence assistant, according to an embodiment of the disclosure;and

FIG. 10 is a view illustrating an electronic device (or user terminal)in a network environment, according to an embodiment of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, with those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to their bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

By the term “include,” “comprise,” and “have”, or “may include,” or “maycomprise” and “may have” used herein indicates disclosed functions,operations, or existence of elements but does not exclude otherfunctions, operations or elements.

For example, the expressions “A or B,” or “at least one of A and/or B”may indicate A and B, A, or B. For instance, the expression “A or B” or“at least one of A and/or B” may indicate (1) at least one A, (2) atleast one B, or (3) both at least one A and at least one B.

The terms such as “1st,” “2nd,” “first,” “second,” and the like usedherein may refer to modifying various different elements of variousembodiments of the disclosure, but are not intended to limit theelements. For instance, “a first user device” and “a second user device”may indicate different users regardless of order or importance. Forexample, a first component may be referred to as a second component andvice versa without departing from the scope of the disclosure.

In an embodiment of the disclosure, it is intended that when a component(for example, a first component) is referred to as being “operatively orcommunicatively coupled with/to” or “connected to” another component(for example, a second component), the component may be directlyconnected to the other component or connected through another component(for example, a third component). In various embodiments of thedisclosure, it is intended that when a component (for example, a firstcomponent) is referred to as being “directly connected to” or “directlyaccessed” another component (for example, a second component), anothercomponent (for example, a third component) does not exist between thecomponent (for example, the first component) and the other component(for example, the second component).

The expression “configured to” used in various embodiments of thedisclosure may be interchangeably used with “suitable for,” “having thecapacity to,” “designed to,” “adapted to,” “made to,” or “capable of”according to the situation, for example. The term “configured to” maynot necessarily indicate “specifically designed to” in terms ofhardware. Instead, the expression “a device configured to” in somesituations may indicate that the device and another device or part are“capable of” For example, the expression “a processor configured toperform A, B, and C” may indicate a dedicated processor (for example, anembedded processor) for performing a corresponding operation or ageneral purpose processor (for example, a central processing unit (CPU)or application processor (AP)) for performing corresponding operationsby executing at least one software program stored in a memory device.

Terms used in various embodiments of the disclosure are used to describecertain embodiments of the disclosure, but are not intended to limit thescope of other embodiments. The terms of a singular form may includeplural forms unless they have a clearly different meaning in thecontext. Otherwise, all terms used herein may have the same meaningsthat are generally understood by a person skilled in the art. Ingeneral, terms defined in a dictionary should be considered to have thesame meanings as the contextual meaning of the related art, and, unlessclearly defined herein, should not be understood differently or ashaving an excessively formal meaning In any case, even the terms definedin the specification are not intended to be interpreted as excludingembodiments of the disclosure.

An electronic device according to various embodiments of the disclosuremay include at least one of a smailphone, a tablet personal computer(PC), a mobile phone, a video telephone, an electronic book reader, adesktop PC, a laptop PC, a netbook computer, a workstation, a server, apersonal digital assistant (PDA), a portable multimedia player (PMP), amotion picture experts group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3)player, a mobile medical device, a camera, or a wearable device. Thewearable device may include at least one of an accessory-type device(e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, acontact lens, a head-mounted device (HMD)), a textile- orclothing-integrated-type device (e.g., an electronic apparel), abody-attached-type device (e.g., a skin pad or a tattoo), or abio-implantable-type device (e.g., an implantable circuit)

In some various embodiments of the disclosure, an electronic device maybe a home appliance. The smart home appliance may include at least oneof, for example, a television (TV), a digital video/versatile disc (DVD)player, an audio, a refrigerator, an air conditioner, a cleaner, anoven, a microwave oven, a washing machine, an air cleaner, a set-topbox, a home automation control panel, a security control panel, a TV box(e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console(e.g., Xbox™ or PlayStation™), an electronic dictionary, an electronickey, a camcorder, or an electronic picture frame

In other various embodiments of the disclosure, an electronic device mayinclude at least one of various medical devices (e.g., various portablemedical measurement devices (e.g., a blood glucose measuring device, aheart rate measuring device, a blood pressure measuring device, a bodytemperature measuring device, or the like), a magnetic resonanceangiography (MRA), a magnetic resonance imaging (MRI), a computedtomography (CT), a scanner, an ultrasonic device, or the like), anavigation device, a global navigation satellite system (GNSS), an eventdata recorder (EDR), a flight data recorder (FDR), a vehicleinfotainment device, electronic equipment for vessels (e.g., anavigation system, a gyrocompass, or the like), avionics, a securitydevice, a head unit for a vehicle, an industrial or home robot, anautomatic teller machine (ATM), a point of sales (POS) device of astore, or an Internet of things (IoT) device (e.g., a light bulb,various sensors, an electric or gas meter, a sprinkler, a fire alarm, athermostat, a streetlamp, a toaster, exercise equipment, a hot watertank, a heater, a boiler, or the like).

According to an embodiment of the disclosure, an electronic device mayinclude at least one of a part of furniture or a building/structure, anelectronic board, an electronic signature receiving device, a projector,or a measuring instrument (e.g., a water meter, an electricity meter, agas meter, a wave meter, or the like). An electronic device may be oneor more combinations of the above-mentioned devices. An electronicdevice according to some various embodiments of the disclosure may be aflexible device. An electronic device according to an embodiment of thedisclosure is not limited to the above-mentioned devices, and mayinclude new electronic devices with the development of new technology.

Hereinafter, an electronic device according to various embodiments ofthe disclosure will be described in more detail with reference to theaccompanying drawings. The term “user” used herein may refer to a personwho uses an electronic device or may refer to a device (e.g., anartificial intelligence electronic device) that uses an electronicdevice.

Prior to describing various embodiments of the disclosure, an integratedintelligent system to which various embodiments of the disclosure iscapable of being applied will be described with reference to FIGS. 1A to1E.

FIG. 1A is a block diagram illustrating an integrated intelligentsystem, according to an embodiment of the disclosure.

Referring to FIG. 1A, an integrated intelligent system 10 may include auser terminal 100, an intelligence server 200, a personal informationserver 300, or a suggestion server 400.

The user terminal 100 may provide a service necessary for a user throughan app (or an application program) (e.g., an alarm app, a message app, apicture (gallery) app, or the like) stored in the user terminal 100. Forexample, the user terminal 100 may execute and operate other app throughan intelligence app (or a speech recognition app) stored in the userterminal 100. A user input for launching and operating the other appthrough the intelligence app of the user terminal 100 may be received.For example, the user input may be received through a physical button, atouch pad, a voice input, a remote input, or the like. According to anembodiment, various types of terminal devices (or an electronic device),which are connected with Internet, such as a mobile phone, a smartphone,PDA, a notebook computer, and the like may be the user terminal 100.According to an embodiment, the user terminal 100 may receive userutterance as a user input. The user terminal 100 may receive the userutterance and may generate an instruction for operating an app based onthe user utterance. As such, the user terminal 100 may operate the appby using the instruction.

The intelligence server 200 may receive a voice input of a user from theuser terminal 100 over a communication network and may change the voiceinput to text data. In another embodiment, the intelligence server 200may generate (or select) a path rule based on the text data. The pathrule may include information about an action (or an operation) forperforming the function of an app or information about a parameternecessary to perform the action. In addition, the path rule may includethe sequence of actions of the app (or the sequence of states). The userterminal 100 may receive the path rule, may select an app depending onthe path rule, and may execute an action included in the path rule inthe selected app. For example, the user terminal 100 may execute theaction and may display a screen corresponding to a state of the userterminal 100, which executes the action, in a display. For anotherexample, the user terminal 100 may execute the action and may notdisplay the result obtained by executing the action in the display. Forexample, the user terminal 100 may execute a plurality of actions andmay display only the result of a part of the plurality of actions in thedisplay. For example, the user terminal 100 may display only the result,which is obtained by executing the last action among a plurality ofactions, in the display. For another example, the user terminal 100 maydisplay the result obtained by executing the action in the display, inresponse to the user input.

A personal information server 300 may include user information or adatabase in which information about a user terminal 100 is stored. Forexample, the personal information server 300 may receive the userinformation (e.g., context information, name information, ageinformation, gender information, address information, occupationinformation, health information, financial information, user preferenceinformation or the like) from the user terminal 100 to store the userinformation in the database. Alternatively, the personal informationserver 300 may receive usage information (e.g., app installationinformation, app execution information, call information, batteryinformation, location information, or communication information) of theuser terminal 100 from the user terminal 100 to store the usageinformation in the database. In an embodiment, in the case where thepersonal information server 300 verifies information received from theuser terminal 100 or information pre-stored in the database, thepersonal information server 300 may update the database.

An intelligence server 200 may be used to receive the user informationor information of the user terminal 100 from the personal informationserver 300 over the communication network and to generate a path ruleassociated with the user input. According to an embodiment, the userterminal 100 may receive the user information from the personalinformation server 300 over the communication network, and may use theuser information as information for managing the database.

The suggestion server 400 may include a database storing information about a function in a terminal, introduction of an application, or afunction to be provided. For example, the suggestion server 400 mayinclude a database associated with a fun ction that a user utilizes byreceiving the user information of the user terminal 100 fro m thepersonal information server 300. The user terminal 100 may receiveinformation about the function to be provided from the suggestion server400 over the communicat ion network and may provide the receivedinformation to the user.

FIG. 1B is a block diagram illustrating a user terminal of an integratedintelligent system, according to an embodiment of the disclosure.

Referring to FIG. 1B, the user terminal 100 may include an input module110, a display 120, a speaker 130, a memory 140, a processor 150, or acommunication circuit 160. Some elements (e.g., 110, 120, 130, 140, or160) of the user terminal 100 may be electrically connected to theprocessor 150. The user terminal 100 may further include a housing, andelements of the user terminal 100 may be seated in the housing or may bepositioned on the housing. In various embodiments, the user terminal 100may be referred to as an “electronic device (or user device)”. Inaddition, the user terminal 100 may not include at least one of theabove-described elements or may further include any other element(s).For example, the user terminal 100 may include elements of an electronicdevice 901 illustrated in FIG. 9 or may include elements of anelectronic device 1001 illustrated in FIG. 10.

According to an embodiment, the input module 110 may receive a userinput from a user. For example, the input module 110 may receive theuser input from the connected external device (e.g., a keyboard or aheadset). For another example, the input module 110 may include a touchscreen (e.g., a touch screen display) coupled to the display 120. Foranother example, the input module 110 may include a hardware key (or aphysical key) placed in the user terminal 100 (or the housing of theuser terminal 100). According to an embodiment, the input module 110 mayinclude a microphone 111 that is capable of receiving the speech of theuser as a voice signal. For example, the input module 110 may include aspeech input system and may receive the speech of the user as the voicesignal through the speech input system. In an embodiment, the microphone111 may be controlled to be always driven (e.g., always on) to receivean input according to a user's speech or may be driven in the case wherea user manipulation is applied to a hardware key (e.g., 112 in FIG. 1C)provided to one area of the user terminal 100.

According to an embodiment, the display 120 may display an image, avideo, and/or an execution screen of an application. For example, thedisplay 120 may display a graphic user interface (GUI) of an app. In anembodiment, at least part of the display 120 may be exposed through onearea of the housing.

According to an embodiment, the speaker 130 may output a sound signal.For example, the speaker 130 may output the sound signal generated inthe user terminal 100 or the sound signal received from an externaldevice to the outside.

According to an embodiment, the memory 140 may store a plurality of apps141 and 143. The plurality of apps 141 and 143 stored in the memory 140may be selected, launched, and executed depending on the user input. Theplurality of apps 141 and 143 may include an application (e.g., a photoapp, a music app, a calendar app, a message app, a call app, or thelike) supporting the function execution of the user terminal 100 and anintelligence app that supports the operation of the speech recognitionservice.

According to an embodiment, the memory 140 may include a databasecapable of storing information necessary to recognize the user input.For example, the memory 140 may include a log database capable ofstoring log information. For another example, the memory 140 may includea persona database capable of storing user information.

According to an embodiment, the memory 140 may store the plurality ofapps 141 and 143, and the plurality of apps 141 and 143 may be loaded tooperate. For example, the plurality of apps 141 and 143 stored in thememory 140 may be loaded by an execution manager module 153 of theprocessor 150 to operate. The plurality of apps 141 and 143 may includeexecution services 141 a and 143 a performing a function or a pluralityof actions (or unit actions) 141 b and 143 b. The execution services 141a and 143 a may be generated by the execution manager module 153 of theprocessor 150 and then may execute the plurality of actions 141 b and143 b.

According to an embodiment, when the actions 141 b and 143 b of the apps141 and 143 are executed, an execution state screen according to theexecution of the actions 141 b and 143 b may be displayed in the display120. For example, the execution state screen may be a screen in a statewhere the actions 141 b and 143 b are completed. For another example,the execution state screen may be a screen in a state where theexecution of the actions 141 b and 143 b is in partial landing (e.g., inthe case where a parameter necessary for the actions 141 b and 143 b arenot input).

According to an embodiment, the execution services 141 a and 143 a mayexecute the actions 141 b and 143 b depending on a path rule. Forexample, the execution services 141 a and 143 a may be activated by theexecution manager module 153, may receive an execution request from theexecution manager module 153 depending on the path rule, and may executethe actions 141 b and 143 b of the apps 141 and 143 depending on theexecution request. If the execution of the actions 141 b and 143 b iscompleted, the execution services 141 a and 143 a may transmitcompletion information to the execution manager module 153.

According to an embodiment, in the case where the plurality of theactions 141 b and 143 b are respectively executed in the apps 141 and143, the plurality of the actions 141 b and 143 b may be sequentiallyexecuted. If the execution of one action (action 1) is completed, theexecution services 141 a and 143 a may open the next action (action 2)and may transmit completion information to the execution manager module153. Here, it is understood that opening an arbitrary action is tochange a state of the arbitrary action to an executable state or toprepare the execution of the arbitrary action. In other words, if thearbitrary action is not opened, the corresponding action may be notexecuted. If the completion information is received, the executionmanager module 153 may transmit an execution request for the nextactions 141 b and 143 b to an execution service (e.g., action 2).According to an embodiment, in the case where the plurality of apps 141and 143 are executed, the plurality of apps 141 and 143 may besequentially executed. For example, if receiving the completioninformation after the execution of the last action of the first app 141is executed, the execution manager module 153 may transmit the executionrequest of the first action of the second app 143 to the executionservice 143 a.

According to an embodiment, in the case where the plurality of theactions 141 b and 143 b are executed in the apps 141 and 143, a resultscreen according to the execution of each of the executed plurality ofthe actions 141 b and 143 b may be displayed in the display 120.According to an embodiment, only a part of a plurality of result screensaccording to the executed plurality of the actions 141 b and 143 b maybe displayed in the display 120.

According to an embodiment, the memory 140 may store an intelligence app(e.g., a speech recognition app) operating in conjunction with anintelligence agent 151. The app operating in conjunction with theintelligence agent 151 may receive and process the utterance of the useras a voice signal. According to an embodiment, the app operating inconjunction with the intelligence agent 151 may be operated by aspecific input (e.g., an input through a hardware key, an input througha touch screen, or a specific voice input) input through the inputmodule 110.

According to an embodiment, the processor 150 may control overallactions of the user terminal 100. For example, the processor 150 maycontrol the input module 110 to receive the user input. The processor150 may control the display 120 to display an image. The processor 150may control the speaker 130 to output the voice signal. The processor150 may control the memory 140 to read or store necessary information.

According to an embodiment, the processor 150 may include theintelligence agent 151, the execution manager module 153, or anintelligence service module 155. In an embodiment, the processor 150 maydrive the intelligence agent 151, the execution manager module 153, orthe intelligence service module 155 by executing instructions stored inthe memory 140. Modules described in various embodiments of thedisclosure may be implemented by hardware or by software. In variousembodiments of the disclosure, it is understood that the action executedby the intelligence agent 151, the execution manager module 153, or theintelligence service module 155 is an action executed by the processor150.

According to an embodiment, the intelligence agent 151 may generate aninstruction for operating an app based on the voice signal received asthe user input. According to an embodiment, the execution manager module153 may receive the generated instruction from the intelligence agent151, and may select, launch, and operate the apps 141 and 143 stored inthe memory 140. According to an embodiment, the intelligence servicemodule 155 may manage information of the user and may use theinformation of the user to process the user input.

The intelligence agent 151 may transmit and process the user inputreceived through the input module 110 to the intelligence server 200.According to an embodiment, before transmitting the user input to theintelligence server 200, the intelligence agent 151 may pre-process theuser input. According to an embodiment, to pre-process the user input,the intelligence agent 151 may include an adaptive echo canceller (AEC)module, a noise suppression (NS) module, an end-point detection (EPD)module, or an automatic gain control (AGC) module. The AEC may remove anecho included in the user input. The NS module may suppress a backgroundnoise included in the user input. The EPD module may detect an end-pointof a user voice included in the user input to search for a part in whichthe user voice is present. The AGC module may adjust the volume of theuser input so as to be suitable to recognize and process the user input.According to an embodiment, the intelligence agent 151 may include allthe pre-processing elements for performance. However, in anotherembodiment, the intelligence agent 151 may include a part of thepre-processing elements to operate at low power.

According to an embodiment, an intelligence agent 151 may include awakeup recognition module recognizing a call of a user. The wakeuprecognition module may recognize a wakeup instruction of the userthrough the speech recognition module. In the case where the wakeuprecognition module receives the wakeup instruction, the wakeuprecognition module may activate the intelligence agent 151 to receivethe user input. According to an embodiment, the wakeup recognitionmodule of the intelligence agent 151 may be implemented with a low-powerprocessor (e.g., a processor included in an audio codec). According toan embodiment, the intelligence agent 151 may be activated depending onthe user input entered through a hardware key. In the case where theintelligence agent 151 is activated, an intelligence app (e.g., a speechrecognition app) operating in conjunction with the intelligence agent151 may be executed. In various embodiments, the wakeup recognitionmodule may be included in an automatic speech recognition module 210 ofFIG. 1D of the intelligence server 200.

According to an embodiment, the intelligence agent 151 may include aspeech recognition module for performing the user input. The speechrecognition module may recognize the user input for executing an actionin an app. For example, the speech recognition module may recognize alimited user (voice) input (e.g., utterance such as “click” forexecuting a capturing action when a camera app is being executed) forexecuting an action such as the wake up instruction in the apps 141 and143. For example, the speech recognition module for recognizing a userinput while assisting the intelligence server 200 may recognize andrapidly process a user instruction capable of being processed in theuser terminal 100. According to an embodiment, the speech recognitionmodule for executing the user input of the intelligence agent 151 may beimplemented in an app processor.

According to an embodiment, the speech recognition module (including thespeech recognition module of a wake up module) of the intelligence agent151 may recognize the user input by using an algorithm for recognizing avoice. For example, the algorithm for recognizing the voice may be atleast one of a hidden markov model (HMM) algorithm, an artificial neuralnetwork (ANN) algorithm, or a dynamic time warping (DTW) algorithm

According to an embodiment, the intelligence agent 151 may change thevoice input of the user to text data. According to an embodiment, theintelligence agent 151 may transmit the voice of the user to theintelligence server 200 to receive the changed text data. As such, theintelligence agent 151 may display the text data in the display 120.

According to an embodiment, the intelligence agent 151 may receive apath rule from the intelligence server 200. According to an embodiment,the intelligence agent 151 may transmit the path rule to the executionmanager module 153.

According to an embodiment, the intelligence agent 151 may transmit theexecution result log according to the path rule received from theintelligence server 200 to the intelligence service module 155, and thetransmitted execution result log may be accumulated and managed inpreference information of the user of a persona module 155 b.

According to an embodiment, the execution manager module 153 may receivethe path rule from the intelligence agent 151 to execute the apps 141and 143 and may allow the apps 141 and 143 to execute the actions 141 band 143 b included in the path rule. For example, the execution managermodule 153 may transmit instruction information for executing theactions 141 b and 143 b to the apps 141 and 143 and may receivecompletion information of the actions 141 b and 143 b from the apps 141and 143.

According to an embodiment, the execution manager module 153 maytransmit or receive the instruction information for executing theactions 141 b and 143 b of the apps 141 and 143 between the intelligenceagent 151 and the apps 141 and 143. The execution manager module 153 maybind the apps 141 and 143 to be executed depending on the path rule andmay transmit the instruction information of the actions 141 b and 143 bincluded in the path rule to the apps 141 and 143. For example, theexecution manager module 153 may sequentially transmit the actions 141 band 143 b included in the path rule to the apps 141 and 143 and maysequentially execute the actions 141 b and 143 b of the apps 141 and 143depending on the path rule.

According to an embodiment, the execution manager module 153 may manageexecution states of the actions 141 b and 143 b of the apps 141 and 143.For example, the execution manager module 153 may receive informationabout the execution states of the actions 141 b and 143 b from the apps141 and 143. For example, in the case where the execution states of theactions 141 b and 143 b are in partial landing (e.g., in the case wherea parameter necessary for the actions 141 b and 143 b are not input),the execution manager module 153 may transmit information about thepartial landing to the intelligence agent 151. The intelligence agent151 may make a request for an input of necessary information (e.g.,parameter information) to the user by using the received information.For another example, in the case where the execution state of theactions 141 b and 143 b are in an operating state, the utterance may bereceived from the user, and the execution manager module 153 maytransmit information about the apps 141 and 143 being executed and theexecution states of the apps 141 and 143 to the intelligence agent 151.The intelligence agent 151 may receive parameter information of theutterance of the user through the intelligence server 200 and maytransmit the received parameter information to the execution managermodule 153. The execution manager module 153 may change a parameter ofeach of the actions 141 b and 143 b to a new parameter by using thereceived parameter information.

According to an embodiment, the execution manager module 153 maytransmit the parameter information included in the path rule to the apps141 and 143. In the case where the plurality of apps 141 and 143 aresequentially executed depending on the path rule, the execution managermodule 153 may transmit the parameter information included in the pathrule from one app to another app.

According to an embodiment, the execution manager module 153 may receivea plurality of path rules. The execution manager module 153 may select aplurality of path rules based on the utterance of the user. For example,in the case where the user utterance specifies the app 141 executing apart of the action 141 b but does not specify the app 143 executing anyother action 143 b, the execution manager module 153 may receive aplurality of different path rules in which the same app 141 (e.g., agallery app) executing the part of the action 141 b is executed and inwhich different apps 143 (e.g., a message app or a Telegram app)executing the other action 143 b. For example, the execution managermodule 153 may execute the same actions 141 b and 143 b (e.g., the samesuccessive actions 141 b and 143 b) of the plurality of path rules. Inthe case where the execution manager module 153 executes the sameaction, the execution manager module 153 may display a state screen forselecting the different apps 141 and 143 included in the plurality ofpath rules in the display 120.

According to an embodiment, the intelligence service module 155 mayinclude a context module 155 a, a persona module 155 b, or a suggestionmodule 155 c.

The context module 155 a may collect current states of the apps 141 and143 from the apps 141 and 143. For example, the context module 155 a mayreceive context information indicating the current states of the apps141 and 143 to collect the current states of the apps 141 and 143.

The persona module 155 b may manage personal information of the userutilizing the user terminal 100. For example, the persona module 155 bmay collect the usage information and the execution result of the userterminal 100 to manage personal information of the user.

The suggestion module 155 c may predict the intent of the user torecommend an instruction to the user. For example, the suggestion module155 c may recommend an instruction to the user in consideration of thecurrent state (e.g., a time, a place, context, or an app) of the user.

A communication circuit 160 (or communication module) according to anembodiment may establish wired communication or wireless communicationaccording to the defined protocol with at least one external device(e.g., intelligence server 200, the personal information server 300 or asuggestion server 400) of an integrated intelligent system 10. Thecommunication circuit 160 may transmit or receive at least oneinformation associated with the operation of the speech recognitionservice based on the wired communication or the wireless communication.

FIG. 1C is a view for describing how an intelligence app of a userterminal is executed, according to an embodiment of the disclosure.

FIG. 1C illustrates that the user terminal 100 receives a user input toexecute an intelligence app (e.g., a speech recognition app) operatingin conjunction with the intelligence agent 151.

According to an embodiment, the user terminal 100 may execute theintelligence app for recognizing a voice through a hardware key 112. Forexample, in the case where the user terminal 100 receives the user inputthrough the hardware key 112, the user terminal 100 may display a UI 121of the intelligence app in the display 120. For example, a user maytouch a speech recognition button 121 a of the UI 121 of theintelligence app for the purpose of entering a voice 111 b in a statewhere the UI 121 of the intelligence app is displayed in the display120. For another example, while continuously pressing the hardware key112 to enter the voice 111 b, the user may enter the voice 111 b.

According to an embodiment, the user terminal 100 may execute theintelligence app for recognizing a voice through the microphone 111. Forexample, in the case where a specified voice 111 a (e.g., wake up!) isentered through the microphone 111, the user terminal 100 may display aUI 121 of the intelligence app in the display 120. In this regard, theabove-described wakeup recognition module may activate the intelligenceagent (151 of FIG. 1B) in association with the specified voice input,and the activation of the intelligence agent 151 may accompany theexecution of the interlocked intelligence app. Furthermore, theexecution of the intelligence app may accompany the activation of anartificial intelligence assistant (e.g., Bixby) interacting (e.g.,dialogue) with a user, based on a specified interface (e.g., dialogueinterface).

FIG. 1D is a block diagram illustrating an intelligence server of anintegrated intelligent system, according to an embodiment of thedisclosure.

Referring to FIG. 1D, the intelligence server 200 may include anautomatic speech recognition (ASR) module 210, a natural languageunderstanding (NLU) module 220, a path planner module 230, a dialoguemanager (DM) module 240, a natural language generator (NLG) module 250,or a text to speech (TTS) module 260. The element 210, 220, 230, 240,250, or 260 of the above-described intelligence server 200 may beimplemented individually, or at least some of the elements may beintegrated. In an embodiment, intelligence server 200 may include acontroller (or a processor), which generally controls the functionoperation of the element 210, 220, 230, 240, 250, or 260, and acommunication interface (or a communication module) supportingcommunication network access. Moreover, the intelligence server 200 mayinclude a storage device (or a memory) including the element 210, 220,230, 240, 250, or 260.

The NLU module 220 or the path planner module 230 of the intelligenceserver 200 may generate a path rule.

According to an embodiment, the ASR module 210 may convert the userinput received from the user terminal 100 to text data. For example, theASR module 210 may include a speech recognition module. The speechrecognition module may include an acoustic model and a language model.For example, the acoustic model may include information associated witha speech, and the language model may include unit phoneme informationand information about a combination of unit phoneme information. Thespeech recognition module may change user speech to text data by usingthe information associated with speech and unit phoneme information. Forexample, the information about the acoustic model and the language modelmay be stored in an automatic speech recognition database (ASR DB) 211.In an embodiment, the ASR module 210 may generate a speaker-dependentrecognition model based on a user input that is received first and maystore the generated model in the database 211. According to anembodiment, the ASR module 210 may determine whether a user is a speakerregistered in the model, with respect to a user input based on a speakerrecognition model.

According to an embodiment, the NLU module 220 may grasp user intent byperforming syntactic analysis or semantic analysis. The syntacticanalysis may divide the user input into syntactic units (e.g., words,phrases, morphemes, and the like) and determine which syntactic elementsthe divided units have. The semantic analysis may be performed by usingsemantic matching, rule matching, formula matching, or the like. Assuch, the NLU module 220 may obtain a domain, intent, or a parameter (ora slot) necessary for the user input to express the intent.

According to an embodiment, the NLU module 220 may determine the intentof the user and parameter by using a matching rule that is divided intoa domain, intent, and a parameter (or a slot) necessary to grasp theintent. For example, the one domain (e.g., an alarm) may include aplurality of intent (e.g., alarm settings, alarm cancellation, and thelike), and one intent may include a plurality of parameters (e.g., atime, the number of iterations, an alarm sound, and the like). Forexample, the plurality of rules may include one or more necessaryparameters. The matching rule may be stored in a natural languageunderstanding database (NLU DB) 221.

According to an embodiment, the NLU module 220 may grasp the meaning ofwords extracted from a user input by using linguistic features (e.g.,grammatical elements) such as morphemes, phrases, and the like and maymatch the meaning of the grasped words to the domain and intent todetermine user intent. For example, the NLU module 220 may calculate howmany words extracted from the user input is included in each of thedomain and the intent, for the purpose of determining the user intent.According to an embodiment, the NLU module 220 may determine a parameterof the user input by using the words that are the basis for grasping theintent. According to an embodiment, the NLU module 220 may determine theuser intent by using the NLU DB 221 storing the linguistic features forgrasping the intent of the user input. According to another embodiment,the NLU module 220 may determine the user intent by using a personallanguage model (PLM). For example, the NLU module 220 may determine theuser intent by using the personalized information (e.g., a contact listor a music list). For example, the PLM may be stored in the NLU DB 221.According to an embodiment, the ASR module 210 as well as the NLU module220 may recognize the voice of the user with reference to the PLM storedin the NLU DB 221.

According to an embodiment, the NLU module 220 may generate a path rulebased on the intent of the user input and the parameter. For example,the NLU module 220 may select an app to be executed, based on the intentof the user input and may determine an action to be executed, in theselected app. The NLU module 220 may determine the parametercorresponding to the determined action to generate the path rule.According to an embodiment, the path rule generated by the NLU module220 may include information about the app to be executed, the action tobe executed in the app, and a parameter necessary to execute the action.

According to an embodiment, the NLU module 220 may generate one pathrule, or a plurality of path rules based on the intent of the user inputand the parameter. For example, the NLU module 220 may receive a pathrule set corresponding to the user terminal 100 from the path plannermodule 230 and may map the intent of the user input and the parameter tothe received path rule set for the purpose of determining the path rule.

According to another embodiment, the NLU module 220 may determine theapp to be executed, the action to be executed in the app, and aparameter necessary to execute the action based on the intent of theuser input and the parameter for the purpose of generating one path ruleor a plurality of path rules. For example, the NLU module 220 mayarrange the app to be executed and the action to be executed in the appby using information of the user terminal 100 depending on the intent ofthe user input in the form of ontology or a graph model for the purposeof generating the path rule. For example, the generated path rule may bestored in a path rule database (PR DB) 231 through the path plannermodule 230. The generated path rule may be added to a path rule set ofthe PR DB 231.

According to an embodiment, the NLU module 220 may select at least onepath rule of the generated plurality of path rules. For example, the NLUmodule 220 may select an optimal path rule of the plurality of pathrules. For another example, in the case where only a part of action isspecified based on the user utterance, the NLU module 220 may select aplurality of path rules. The NLU module 220 may determine one path ruleof the plurality of path rules depending on an additional input of theuser.

According to an embodiment, the NLU module 220 may transmit the pathrule to the user terminal 100 in response to a request for the userinput. For example, the NLU module 220 may transmit one path rulecorresponding to the user input to the user terminal 100. For anotherexample, the NLU module 220 may transmit the plurality of path rulescorresponding to the user input to the user terminal 100. For example,in the case where only a part of action is specified based on the userutterance, the plurality of path rules may be generated by the NLUmodule 220.

According to an embodiment, the path planner module 230 may select atleast one path rule of the plurality of path rules.

According to an embodiment, the path planner module 230 may transmit apath rule set including the plurality of path rules to the NLU module220. The plurality of path rules of the path rule set may be stored inthe PR DB 231 connected to the path planner module 230 in the tableform. For example, the path planner module 230 may transmit a path ruleset corresponding to information (e.g., OS information or appinformation) of the user terminal 100, which is received from theintelligence agent 151, to the NLU module 220. For example, a tablestored in the PR DB 231 may be stored for each domain or for eachversion of the domain.

According to an embodiment, the path planner module 230 may select onepath rule or the plurality of path rules from the path rule set totransmit the selected one path rule or the selected plurality of pathrules to the NLU module 220. For example, the path planner module 230may match the user intent and the parameter to the path rule setcorresponding to the user terminal 100 to select one path rule or aplurality of path rules and may transmit the selected one path rule orthe selected plurality of path rules to the NLU module 220.

According to an embodiment, the path planner module 230 may generate theone path rule or the plurality of path rules by using the user intentand the parameter. For example, the path planner module 230 maydetermine the app to be executed and the action to be executed in theapp based on the user intent and the parameter for the purpose ofgenerating the one path rule or the plurality of path rules. Accordingto an embodiment, the path planner module 230 may store the generatedpath rule in the PR DB 231.

According to an embodiment, the path planner module 230 may store thepath rule generated by the NLU module 220 in the PR DB 231. Thegenerated path rule may be added to the path rule set stored in the PRDB 231.

According to an embodiment, the table stored in the PR DB 231 mayinclude a plurality of path rules or a plurality of path rule sets. Theplurality of path rules or the plurality of path rule sets may reflectthe kind, version, type, or characteristic of a device performing eachpath rule.

According to an embodiment, the DM module 240 may determine whether theuser intent grasped by the NLU module 220 is clear. For example, the DMmodule 240 may determine whether the user intent is clear, based onwhether the information of a parameter is sufficient. The DM module 240may determine whether the parameter grasped by the NLU module 220 issufficient to perform a task. According to an embodiment, in the casewhere the user intent is not clear, the DM module 240 may perform afeedback for making a request for necessary information to the user. Forexample, the DM module 240 may perform a feedback for making a requestfor information about the parameter for grasping the user intent.

According to an embodiment, the DM module 240 may include a contentprovider module. In the case where the content provider module executesan action based on the intent and the parameter grasped by the NLUmodule 220, the content provider module may generate the result obtainedby performing a task corresponding to the user input. According to anembodiment, the DM module 240 may transmit the result generated by thecontent provider module as the response to the user input to the userterminal 100.

According to an embodiment, the natural language generating module NLG250 may change specified information to a text form. Information changedto the text form may be a form of a natural language utterance. Forexample, the specified information may be information about anadditional input, information for guiding the completion of an actioncorresponding to the user input, or information for guiding theadditional input of the user (e.g., feedback information about the userinput). The information changed to the text form may be displayed in thedisplay 120 after being transmitted to the user terminal 100 or may bechanged to a voice form after being transmitted to the TTS module 260.

According to an embodiment, the TTS module 260 may change information ofthe text form to information of a voice form. The TTS module 260 mayreceive the information of the text form from the NLG module 250, maychange the information of the text form to the information of a voiceform, and may transmit the information of the voice form to the userterminal 100. The user terminal 100 may output the information of thevoice form to the speaker 130

According to an embodiment, the NLU module 220, the path planner module230, and the DM module 240 may be implemented with one module. Forexample, the NLU module 220, the path planner module 230 and the DMmodule 240 may be implemented with one module, may determine the userintent and the parameter, and may generate a response (e.g., a pathrule) corresponding to the determined user intent and parameter. Assuch, the generated response may be transmitted to the user terminal100.

According to an embodiment, in the case where at least one element(e.g., the ASR module 210, the NLU module 220, or the like) performs theallocated function or interacts with the user terminal 100 (or a user),the above-described intelligence server 200 may update the at least oneelement. For example, the intelligence server 200 may update a model ora database associated with the function execution of the at least oneelement. The intelligence server 200 may transmit update informationabout the at least one element to the user terminal 100.

FIG. 1E is a view illustrating a method in which a NLU module generatesa path rule, according to an embodiment of the disclosure.

Referring to FIG. 1E, according to an embodiment, the NLU module 220 maydivide the function of an app into unit actions (e.g., A to F) and maystore the divided unit actions in the PR DB 231. For example, the NLUmodule 220 may store a path rule set, which includes a plurality of pathrules A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F divided into unitactions, in the PR DB 231.

According to an embodiment, the PR DB 231 of the path planner module 230may store the path rule set for performing the function of an app. Thepath rule set may include a plurality of path rules each of whichincludes a plurality of actions. An action executed depending on aparameter input to each of the plurality of actions may be sequentiallyarranged in the plurality of path rules. According to an embodiment, theplurality of path rules implemented in a form of ontology or a graphmodel may be stored in the PR DB 231.

According to an embodiment, the NLU module 220 may select an optimalpath rule A-B1-C3-D-F of the plurality of path rules A-B1-C1, A-B1-C2,A-B1-C3-D-F, and A-B1-C3-D-E-F corresponding to the intent of a userinput and the parameter.

According to an embodiment, in the case where there is no path rulecompletely matched to the user input, the NLU module 220 may transmit aplurality of rules to the user terminal 100. For example, the NLU module220 may select a path rule (e.g., A-B1) partly corresponding to the userinput. The NLU module 220 may select one or more path rules (e.g.,A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) including the pathrule (e.g., A-B1) partly corresponding to the user input and maytransmit the one or more path rules to the user terminal 100.

According to an embodiment, the NLU module 220 may select one of aplurality of path rules based on an input added by the user terminal 100and may transmit the selected one path rule to the user terminal 100.For example, the NLU module 220 may select one path rule (e.g.,A-B1-C3-D-F) of the plurality of path rules (e.g., A-B1-C1, A-B1-C2,A-B1-C3-D-F, and A-B1-C3-D-E-F) depending on the user input (e.g., aninput for selecting C3) additionally entered by the user terminal 100for the purpose of transmitting the selected one path rule to the userterminal 100.

According to another embodiment, the NLU module 220 may determine theintent of a user and the parameter corresponding to the user input(e.g., an input for selecting C3) additionally entered by the userterminal 100 for the purpose of transmitting the user intent or theparameter to the user terminal 100. The user terminal 100 may select onepath rule (e.g., A-B1-C3-D-F) of the plurality of path rules (e.g.,A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) based on thetransmitted intent or the transmitted parameter.

As such, the user terminal 100 may complete the actions of the apps 141and 143 based on the selected one path rule.

According to an embodiment, in the case where a user input in whichinformation is insufficient is received by the intelligence server 200,the NLU module 220 may generate a path rule partly corresponding to thereceived user input. For example, the NLU module 220 may transmit thepartly corresponding path rule to the intelligence agent 151. Theintelligence agent 151 may transmit the partly corresponding path ruleto the execution manager module 153, and the execution manager module153 may execute the first app 141 depending on the path rule. Theexecution manager module 153 may transmit information about aninsufficient parameter to the intelligence agent 151 while executing thefirst app 141. The intelligence agent 151 may make a request for anadditional input to a user by using the information about theinsufficient parameter. If the additional input is received by the user,the intelligence agent 151 may transmit and process the additional inputto the intelligence server 200. The NLU module 220 may generate a pathrule to be added, based on the intent of the user input additionallyentered and parameter information and may transmit the path rule to beadded, to the intelligence agent 151. The intelligence agent 151 maytransmit the path rule to the execution manager module 153 and mayexecute the second app 143.

According to an embodiment, in the case where a user input, in which aportion of information is missed, is received by the intelligence server200, the NLU module 220 may transmit a user information request to thepersonal information server 300. The personal information server 300 maytransmit information of a user entering the user input stored in apersona database to the NLU module 220. The NLU module 220 may select apath rule corresponding to the user input in which a part of an actionis missed, by using the user information. As such, even though the userinput in which a portion of information is missed is received by theintelligence server 200, the NLU module 220 may make a request for themissed information to receive an additional input or may determine apath rule corresponding to the user input by using user information.

As described above through FIG. 1A to 1E, the integrated intelligentsystem 10 of FIG. 1A may accompany a series of processes for providing aspeech recognition-based service. For example, the user terminal 100 ofFIG. 1A may receive a user's speech as a user input to transmit the userinput to the intelligence server 200 of FIG. 1A, and the intelligenceserver 200 may generate a path rule based on the user input. The userterminal 100 may receive the generated path rule from the intelligenceserver 200 and may execute the function of a specific app depending onthe path rule. In an embodiment, in an operation of performing theabove-described process, the user terminal 100 may calculate a numericalvalue (e.g., experience point) associated with an artificialintelligence assistant (e.g., Bixby) supporting a speech recognitionservice, based on update information about at least one element (e.g.,the ASR module 210 of FIG. 1D or the NLU module 220 of FIG. 1D) of theintelligence server 200 or database update information of the personalinformation server 300 of FIG. 1A. For example, the intelligence agent151 of FIG. 1B of the user terminal 100 may assign the predeterminedpoint to at least one of the ASR module 210, the NLU module 220, and thepersonal information server 300 based on the update information and maycollect the assigned point to calculate the experience point of theartificial intelligence assistant. In various embodiments, theintelligence agent 151 may assign a point to a specified activity (e.g.,activity associated with a speech recognition service or the operationof an intelligence app) that a user performs on the user terminal 100,and may further refer to the activity point to calculate the experiencepoint of the artificial intelligence assistant. In various embodiments,the calculation of the experience point of the user terminal 100 may beperformed by a scoring manager module included in the intelligenceserver 200 by software or hardware, and the user terminal 100 mayreceive information about the experience point from the scoring managermodule.

In an embodiment, the user terminal 100 may request the function of aspeech recognition service corresponding to the experience point, basedon the calculated experience point. In this regard, the intelligenceserver 200 may classify a speech recognition service function to beprovided to the user terminal 100, into a plurality of functions and mayprovide the user terminal 100 with a service function corresponding tothe experience point of the artificial intelligence assistant.Hereinafter, various embodiments associated with the calculation of theexperience point of the artificial intelligence assistant and therefinement of the speech recognition service function of the userterminal 100 will be described.

FIGS. 2A to 2C are views illustrating various interfaces associated witha speech recognition service, according to various embodiments of thedisclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B (or theprocessor 150 of FIG. 1B) of the user terminal 100 may display theexperience point of the artificial intelligence assistant (e.g., Bixby)calculated through a series of processes, through a specified interface.A user may access the interface to verify the experience point and atleast one information associated with the experience point calculation.As such, the user may recognize the degree of trust in the artificialintelligence assistant (or speech recognition service) and may obtaininformation contributing to the improvement of the experience point.

In this regard, referring to FIG. 2A, an intelligence app that isinstalled in the user terminal 100 and supports the operation of thespeech recognition service may output a first interface 1, in responseto user control associated with a specific category (or menu). Anexperience point 3 (e.g., 2100 XP) of an artificial intelligenceassistant and points 5, 7, and 9 (e.g., 1100 Pts, 500 Pts, and 500 Pts)respectively assigned to at least one or more elements (e.g., the ASRmodule 210 of FIG. 1D, the NLU module 220 of FIG. 1D, and the personalinformation server 300 of FIG. 1A) contributing to the calculation ofthe experience point 3 may be displayed in at least one area of thefirst interface 1. In an embodiment, the experience point 3 may becalculated by summing points 5, 7, and 9 respectively assigned to theASR module 210, the NLU module 220, and the personal information server300 or may be applied as the highest point or the lowest point amongpoints. In various embodiments, the points 5, 7, and 9 may be displayedconcurrently with the output of the first interface 1 or may bedisplayed in a specified area (e.g., an icon of the artificialintelligence assistant) on the first interface 1 together with arelevant diagram (e.g., graph) in the case where a user input (e.g.,touch) is applied. In various embodiments, a user activity point (e.g.,a point assigned in the case where a user performs a specified activitythrough the user terminal 100) contributing to the calculation of theexperience point 3 may be further displayed on the first interface 1.

Referring to FIG. 2B, for example, a second interface 13 (e.g., widget)that operates in conjunction with the execution of the intelligence appmay be included on a home screen 11 (or lock screen) of the userterminal 100. At least one of level information, badge information, anduser image that correspond to the experience point of the artificialintelligence assistant may be displayed on the second interface 13. Invarious embodiments, as illustrated in FIG. 2B, the level informationmay be displayed in the form of a number as shown or may be displayed asa name of a specified object. For example, the object may include atleast one planet, and the level information may be expressed as a planet(e.g., Solar system planet (e.g., Mercury, Venus, Earth, Mars, Jupiter,Saturn, Uranus, Neptune, and Pluto), dwarf Planet (Ceres, Haumea,Makemake, or Eris), Sedna, Moon, Galaxy, or the like) of a name that isdifferent depending on the experience point value of an artificialintelligence assistant. For example, as the experience point value ofthe artificial intelligence assistant increases, the level informationmay be expressed as the name of a planet that exists at a great distancebased on the sun. In an embodiment, with regard to the operation of aspeech recognition service, the grade (or level information or planetname) of the level may be determined depending on at least one activityor function operation performed by a user or a part of elements of theintegrated intelligent system 10 of FIG. 1A. For example, the levelinformation of the different grade or the planet name may be determineddepending on the number of times that the at least one activity (e.g.,providing feedback of the user's satisfaction or dissatisfactionattribute, suggesting the user's variation, adopting the variationsuggestion, or the like that is to be described below) is performed. Inthe case where a user input (e.g., touch) is applied to at least onearea of the second interface 13, the second interface 13 may be switchedto a third interface 15 corresponding to at least part of the executionscreen of an intelligence app. In an embodiment, at least one of theexperience point (e.g., 1253 XP) of the artificial intelligenceassistant, the point (e.g., 35 Pts) obtained on the day, the maximumnumber of points obtainable on the day, guide information (e.g.,experience point information assigned in the case where the function ofa specific app is performed based on the execution of a path rule,experience point information assigned in the case where variationsuggestion associated with the operation of the speech recognitionservice occurs from a user, experience point information additionallyassigned in the case where the variation suggestion is adopted, or thelike) associated with the improvement of the experience point, andnotice information associated with the operation of a speech recognitionservice may be included on the third interface 15.

According to an embodiment, a button or a tap 15 a that is implementedto include the level information may be provided on the third interface15; in the case where a user touch input is applied to at least one areaof the button or tap, the third interface 15 may be switched to aninterface 16 as illustrated in FIG. 2C. The details about previouslyestablished level information may be displayed on the switched interface16 in the form of a list, and level information 16a that can beestablished in the future (e.g., level information of the next grade)may be displayed at the top of the list. In various embodiments, thelevel information may be referred to as an indicator of a specificreward or benefits to be provided in operating the speech recognitionservice; in this regard, in the case where a user touch input is appliedto the details about previously established level information, detailsabout the previously provided reward or benefits may be displayed.

FIG. 3A is a view illustrating a speech recognition service operatingmethod of a user terminal, according to an embodiment of the disclosure.

Referring to FIG. 3A, in operation 301, a user may operate a speechrecognition service through the user terminal 100 of FIG. 1B. In thisregard, a user may manipulate a specified hardware key (e.g., thehardware key 112 of FIG. 1C) on a user terminal or may perform aspecified wakeup command speech (e.g., “wake up” or “Hi, Bixby”) toexecute an intelligence app. In an embodiment, the execution of theintelligence app may accompany the activation of an artificialintelligence assistant (e.g., Bixby) responding to a user speech basedon the speech recognition result of the intelligence server 200 of FIG.1D. After the execution of the intelligence app or the activation of theartificial intelligence assistant, the user may perform speech includinga specific command or intent. Alternatively, after the execution of theintelligence app, a user may perform activity associated with theoperation of the speech recognition service.

In operation 303, the intelligence agent 151 of FIG. 1B of a userterminal may determine whether at least one event occurring uponoperating the speech recognition service of a user or operating anintelligence app of a user corresponds to a specified scoring event. Forexample, the specified scoring event may be an event that accompaniesthe element update of an intelligence server or the database update ofthe personal information server 300 of FIG. 1A. Furthermore, it isunderstood that the specified scoring event is an event that thefunction of the speech recognition service operable in a user terminalis updated (or refined) by contributing to the calculation of theexperience point of the artificial intelligence assistant, and thespecified scoring event may be referred to as “first to eleventhembodiments” described through FIGS. 4A to 7D.

According to an embodiment, in the case where the generated event is notassociated with the scoring event, an intelligence agent may exclude thefunction update of the speech recognition service. Alternatively, in thecase where it is determined that the generated at least one event is thescoring event, in operation 305, the intelligence agent may assign apoint to an element (e.g., the ASR module 210 of FIG. 1D, the NLU module220 of FIG. 1D, or the personal information server 300 of FIG. 1A)associated with the generated event or may assign an activity pointthereto. In an embodiment, the intelligence agent may reflect the pointto the calculation of the experience point of an artificial intelligenceassistant. For example, the intelligence agent may calculate theexperience point of the artificial intelligence assistant by summing atleast one or more points or may apply the highest point or the lowestpoint of at least one point to the experience point. In variousembodiments, the experience point is not limited to the above-describedcalculation method or application method, and the experience point maybe calculated through various arithmetic operations based on points.

In operation 307, the intelligence agent may request the intelligenceserver to refine or update the speech recognition service function basedon the experience point of the artificial intelligence assistant and mayselectively operate the service function provided from the intelligenceserver.

FIG. 3B is a view illustrating a speech recognition service operatingmethod of an intelligence server, according to an embodiment of thedisclosure.

Referring to FIG. 3A, the calculation of the experience point of anartificial intelligence assistant according to an embodiment may beperformed by the user terminal 100 of FIG. 1B of the integratedintelligent system 10 of FIG. 1A. Alternatively, as described belowthrough FIG. 3B, the calculation of the experience point may beperformed by the intelligence server 200 of FIG. 1D of the integratedintelligent system 10.

In operation 309, the intelligence server 200 may receive specified datafrom an external device (e.g., the user terminal 100) through acommunication interface (or communication module). For example, theintelligence server may receive data (e.g., a user input according tothe speech of a user of an external device) associated with a voiceinput, from the external device.

In operation 311, the intelligence server may process the received data.In an embodiment, the data processing may include an operation in whichthe ASR module 210 in FIG. 1D included as at least a part of elements ofthe intelligent server recognizes the data and converts the data intotext data. Alternatively, the processing may include an operation inwhich the NLU module 220 of FIG. 1D included in the intelligence serverderives user intent from the data and generates a path rule associatedwith an operation to be performed by the external device, based on thederived user intent. Alternatively, the processing may include anoperation in which at least one of the ASR module 210 and the NLU module220 interacts with the external device (or a user of the externaldevice) to update a relevant model or the database 211 or 221 of FIG.1D, in an operation of performing the function of the above-describedmodules 210 and 220.

In operation 313, the intelligence server may calculate a numericalvalue (e.g., point) associated with at least one of the ASR module 210and the NLU module 220. In this regard, the intelligence server mayinclude a scoring manager module that calculates the numerical value. Inan embodiment, with regard to the operation of a speech recognitionservice of the external device user, in the case where at least one ofthe ASR module 210 and the NLU module 220 performs the assignedfunction, interacts (e.g., query and respond) with the user of theexternal device, or updates the relevant model, the database, or thelike, the scoring manager module may assign the predetermined numericalvalue to the corresponding module. The scoring manager may collect atleast one numerical value to calculate the experience point of theartificial intelligence assistant supporting interaction with theexternal device user. In operation 315, the intelligence server maytransmit information associated with a numerical value or an experiencepoint to the external device through a communication interface. Theintelligence server may refine the function of the speech recognitionservice corresponding to the request to provide the external device withthe refined function of the speech recognition service, depending on therequest of the external device based on the numerical value or theexperience point.

FIG. 4A is a flowchart illustrating a first embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an ASR module, according to an embodiment of thedisclosure.

Referring to FIG. 4A, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of theartificial intelligence assistant based on the amount of information ofa user input according to user speech.

In this regard, referring to FIG. 4A, in operation 401, the intelligenceagent may collect the user input (or a voice signal) according to theuser speech. For example, the intelligence agent may collect the userinput, which has been stored in the memory 140 of FIG. 1B during aspecified time period or which has been transmitted to the ASR module210 of FIG. 1D of the intelligence server 200 of FIG. 1D.

In operation 403, the intelligence agent may extract a user input, ofwhich the speech recognition is successful (or which is converted to aclear text) by the ASR module, of the collected at least one user input.The above-described operation may be implemented by sharing informationabout the speech recognition result between the user terminal and theASR module of an intelligence server.

In operation 405, the intelligence agent may determine the presence orabsence of the speaker-dependently recognized user input of at least oneuser input of which the speech recognition is successful. For example,the intelligence agent may obtain speaker recognition result informationfrom the ASR module 210 of FIG. 1D of the intelligence server 200 ofFIG. 1D to identify the speaker-dependently recognized user input. Inthe case where there is no speaker-dependently user input of the userinput of which the speech recognition is successful, the intelligenceagent may exclude the calculation of the experience point of theartificial intelligence assistant.

In the case where the at least one speaker-dependent user input ispresent, in operation 407, the intelligence agent may calculate theamount of information (e.g., accumulated time) by summing speech timecorresponding to each speaker-dependent user input.

In operation 409, the intelligence agent may assign points Pts to theASR module performing the speech recognition function associated with auser input, based on the amount of calculated information.

Pts _(n) =Pts _(n-1) +αt   Equation 1

Pts_(n): the final point of the ASR module

Pts_(n-1): the previous point of the ASR module

α: the scoring coefficient associated with the ASR module

‘t’: the accumulated time of user speech corresponding to aspeaker-dependent user input

It is understood that Equation 1 is an equation associated with thescoring of the ASR module. The intelligence agent may add the pointPts_(n-1), which has been assigned to the ASR module, to the amount ofcalculated information (e.g., accumulated time ‘t’) to assign the finalpoint Pts_(n) to the ASR module.

In operation 411, the intelligence agent may calculate the experiencepoint of the artificial intelligence assistant by summing the finalpoint Pts_(n) of the ASR module and other points (e.g., the point of theNLU module 220 of FIG. 1D, the point of the personal information server300 of FIG. 1A, a user activity point, or the like). Alternatively, theintelligence agent may calculate the experience point of the artificialintelligence assistant by comparing the final point Pts_(n) of the ASRmodule with the other points based on a specified reference (e.g., thehighest point or the lowest point).

FIG. 4B is a flowchart illustrating a second embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an ASR module, according to an embodiment of thedisclosure.

Referring to FIG. 4B, in operation 413, a user may perform wakeupcommand speech in association with the operation of a speech recognitionservice. It is understood that the wakeup command speech is the speechperformed upon operating the speech recognition service of the user forthe first time. Alternatively, it is understood that the wakeup commandspeech is the speech performed in an operation in which the usercontrols the setting of an intelligence app to designate the wakeupcommand speech. The wakeup command speech of the user may be transmittedto a wakeup recognition module included in the ASR module 210 of FIG. 1Dof the intelligence server 200 of FIG. 1D by the processor 150 of FIG.1B. The wakeup recognition module may generate a wakeup recognitionmodel based on the transmitted wakeup command speech.

In operation 415, the intelligence agent may determine whether toperform the user's training on the wakeup command speech. It isunderstood that the training is the execution of the user's iterativewakeup command speech for mapping the user's wakeup command speech tothe wakeup command recognition model. In an embodiment, the trainingassociated with the wakeup command recognition model may be performedbased on a specified algorithm (e.g., maximum likelihood estimation,gradient descent, linear regression, or the like).

In the case where the training is performed on the generated wakeupcommand recognition model, in operation 417, the intelligence agent mayassign the first point Pts to the ASR module performing speechrecognition on the user speech.

Pts _(n) =Pts _(n-1)+(β·p(x|λ)+β₀)·S _(wakeup)   Equation 2

Pts_(n): the final point of the ASR module

Pts_(n-1): the previous point of the ASR module

S_(wakeup): the unit score of a wakeup command recognition model

p(x|λ): the unit score of an algorithm used for wakeup commandrecognition model.

β or β₀: correction coefficient

It is understood that Equation 2 is another equation associated with thescoring of the ASR module. The intelligence agent may add the unit scoreS_(wakeup) of the wakeup command recognition model, to which algorithmunit score p (x|λ) is reflected, to the point Pts_(n-1), which has beenassigned to the ASR module to assign the final point Pts_(n) to the ASRmodule.

Alternatively, in operation 419, in the case where the training is notperformed on the wakeup command recognition model, the intelligenceagent may assign the second point Pts to the ASR module.

Pts _(n) =Pts _(n-1) +S _(wakesup)   Equation 3

Pts_(n): the final point of the ASR module

Pts_(n-1): the previous point of the ASR module

S_(wakeup): the unit score of a wakeup command recognition model

It is understood that Equation 3 is still another equation associatedwith the scoring of the ASR module. The intelligence agent may add theunit score S_(wakeup) of the generated wakeup command recognition modelto the point Pts_(n-1), which has been assigned to the ASR module toassign the final point Pts_(n) to the ASR module. In this operation, theintelligence agent may proportionally increase the unit score of thewakeup command recognition model based on the number of recognitionmodels generated by the wakeup recognition module, the number of usersrespectively matched to a plurality of recognition models, or the like.

In operation 421, the intelligence agent may calculate the experiencepoint of the artificial intelligence assistant by summing the finalpoint Pts_(n) of the ASR module according to the assignment of the firstpoint or the second point to the ASR module and other points (e.g., thepoint of the NLU module 220 of FIG. 1D, the point of the personalinformation server 300 of FIG. 1A, a user activity point, or the like).Alternatively, the intelligence agent may calculate the experience pointof the artificial intelligence assistant by comparing the final pointPts_(n) of the ASR module with the other points based on a specifiedreference (e.g., the highest point or the lowest point). In variousembodiments, the wakeup recognition module may generate a recognitionmodel associated with various command speeches in addition to theabove-described wakeup command speech; in this case, a point may beassigned to the ASR module as in Equation 2 or Equation 3.

FIG. 4C is a flowchart illustrating a third embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an ASR module, according to an embodiment of thedisclosure.

FIGS. 4D and 4E are views illustrating various embodiments to train anartificial intelligence assistant, according to various embodiments ofthe disclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of anartificial intelligence assistant based on the result revision of speechrecognition by the ASR module 210 of FIG. 1D.

Referring to FIG. 4C, in operation 423, the user may execute anintelligence app supporting the operation of a speech recognitionservice (e.g., manipulating a hardware key on an electronic device orspeaking a specified wakeup command). As such, the artificialintelligence assistant responding to user speech may be activated, and auser may perform speech including a specific command or intent.

In operation 425, the activated artificial intelligence assistant mayrespond to the user speech. The response of the artificial intelligenceassistant may be based on function execution by at least one of the ASRmodule 210 of FIG. 1D, the NLG module 250 of FIG. 1D, and the TTS module260 of FIG. 1D in the intelligence server 200 of FIG. 1D. In anembodiment, the intelligence agent (or the processor 150 of FIG. 1B) mayoutput the response of the artificial intelligence assistant on aninterface (e.g., dialogue interface) supporting interaction between theuser and the artificial intelligence assistant or may output theresponse through the speaker 130 of FIG. 1B.

In operation 427, the user may determine whether an error is present inthe speech recognition result associated with the speech, based on theoutput response of the artificial intelligence assistant. For example,in the case where an error is present in the output text or in the casewhere the response of the artificial intelligence assistant is inconflict with the speech, the user may determine whether the speechrecognition associated with the speech is unclearly performed. In thecase where an event associated with the revision of the speechrecognition result does not occur because speech recognition associatedwith user speech is clearly performed, the intelligence agent mayexclude the calculation of the experience point of the artificialintelligence assistant.

In the case where the user recognizes the uncertainty of the speechrecognition associated with the speech, in operation 429, the user mayperform feedback speech on the response of the artificial intelligenceassistant that is in conflict with a text or speech in which an error ispresent. The feedback speech may be transmitted to the DM module 240 ofFIG. 1D of the intelligence server, and the DM module 240 may provide auser terminal with option information for training the artificialintelligence assistant (or for improving the speech recognition rate ofthe ASR module). The user may perform the option and may train theartificial intelligence assistant in association with speech recognitionthat is unclearly performed. Hereinafter, an example associated withabove-described operation 427 and operation 429 will be described withreference to FIG. 4D.

Referring to a first dialogue interface 17 a of FIG. 4D, in the casewhere the user performs speech (e.g., “send a message to Sungjin”)including a specific command or intent, a user input associated with theuser speech may be transmitted to the ASR module of the intelligenceserver, and the speech recognition may be performed. In this operation,in the case where the user speech is unclear, or in the case where noiseis generated around the user during user speech, an error (e.g., send amessage to Sangjin) may be present in the speech recognition result (orthe converted text) of the ASR module. The intelligence server maytransmit, to the user terminal, text information converted by the ASRmodule and path rule information by the NLU module 220 of FIG. 1D. Theintelligence agent may output the transmitted text information, and anexecution service 141 a or 143 a of FIG. 1B of a specific app (e.g.,message app) corresponding to the path rule may perform at least oneunit operation included in the path rule. At this time, in the casewhere there is no information corresponding to information (e.g.,parameter information) accompanying the execution of the unit operationon a user terminal, the intelligence agent may provide a DM module witha notification thereof The DM module may transmit, to an electronicdevice, text information (e.g., there is no corresponding contact.Please re-enter the recipient.) about the absence of the information andtext information for requesting the re-execution of the user speech.

Referring to a second dialogue interface 17 b, when the user re-performsspeech (e.g., “send a message to Sungjin”), the ASR module may performspeech recognition on the re-performed speech. At this time, in the casewhere the speech recognition result of the re-performed speech is thesame as the previous speech recognition result, the DM module maytransmit, to a user terminal, specified text information (e.g., Do youthink I don't understand well? I can understand better if you let meknow which part is the problem.). Together with the transmission of thespecified text information, the DM module may transmit optioninformation associated with the training of the artificial intelligenceassistant or the improvement of speech recognition rate. For example, itis understood that the option information is a training tap 21 output onthe second dialogue interface 17 b.

Referring to a third dialogue interface 17 c, in the case where the userselects the training tap 21, the DM module may make a request foradditional information associated with the training of the artificialintelligence assistant to the user terminal. For example, the DM modulemay transmit text information (e.g., please select the part requiringthe training) requesting details about the error of the speechrecognition result and list information about at least one or more wordsconstituting the text converted by the ASR module. For example, it isunderstood that the list information includes “to Sangjin” tap 23,“message” tap 25, and “send” tap 27.

Referring to a fourth dialogue interface 17d and a fifth dialogueinterface 17 e, in the case where a user input (e.g., touch) is appliedto a word tap (e.g., “to Sangjin” tap 29) corresponding to the error onthe third dialogue interface 17 c, the DM module may transmit textinformation (e.g., please enter your desired content instead of “toSangjin”) for requesting the revision of the error, to the userterminal. To cope with the issue, an intelligence agent (or a processor)may output an input interface (e.g., SIP keyboard), and the user mayenter details (e.g., to Sungjin) about the revision of the error. Afterreceiving revision information from the user terminal, the DM module maytransmit at least one text information (e.g., please tell “to Sungjin”,please tell “this to Sungjin”, please tell “send a message to Sungjin”,or the like) associated with a speech request including the revision. Inthe case where the user performs the requested speech, the ASR modulemay update a speech recognition model based on a user input according touser speech (e.g., “to Sungjin”, “this to Sungjin” and “send a messageto Sungjin”).

FIG. 4E is a diagram illustrating another example associated withoperation 427 and operation 429, and one example described in FIG. 4Emay include a training process similar to an example described in FIG.4D. Accordingly, a duplicated description may be omitted.

The description of a sixth dialogue interface 17 f of FIG. 4E maycorrespond to that described through the first dialogue interface 17 aof FIG. 4D. However, according to an embodiment, in the case where textinformation (e.g., there is no corresponding contact. Please re-enterthe recipient.) about the absence of the information and textinformation for requesting the re-execution of the user speech is outputon a sixth dialogue interface 17 f, the user may apply a user input(e.g., touch) to the text information 31 (e.g., send a message toSangjin) converted by the ASR module.

Referring to a seventh dialogue interface 17 g, the DM module of anintelligence server may transmit at least one option informationassociated with the training of the artificial intelligence assistant orthe improvement of the speech recognition rate, in response to the userinput. For example, it is understood that the at least one optioninformation includes the training tap 33, “to Sungjin” tap 35, and “toSeungjin” tap 37. The user may apply a user input to a tap, which isassociated with the error, in at least one option information output onthe seventh dialogue interface 17 g to designate the requestedrecipient. Alternatively, in the case where there is no tap associatedwith the error on at least one option information, the user may selectthe training tap 33 to proceed to the training process as illustrated inan eighth dialogue interface 17 h and a ninth dialogue interface 17 i.The descriptions of the eighth dialogue interface 17 h and the ninthdialogue interface 17 i may be the same as or correspond to thatdescribed through the fourth dialogue interface 17d and the fifthdialogue interface 17 e in FIG. 4D.

Returning to FIG. 4C, in operation 431, in a process to revise theabove-described speech recognition result, the intelligence agent of auser terminal may assign the point Pts to the ASR module, based oninformation (e.g., the number of revised words) about the revision.

Ptsn=Pts_(n-1)+N_(W)S_(W)   Equation 4

Pts_(n): the final point of the ASR module

Pts_(n-1): the previous point of the ASR module

N_(W): the number of revised words

S_(W): the unit score of the revised word

It is understood that Equation 4 is yet another equation associated withthe scoring of the ASR module. The intelligence agent may add a value,which is obtained by multiplying the number of revised words Nw by theunit score S_(W), to the point Pts_(n-1), which has been assigned to theASR module to assign the final point Pts_(n) to the ASR module.

Operation 433 may be the same as or correspond to operation 411described through FIG. 4A.

FIG. 5A is a flowchart illustrating a fourth embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an NLU module, according to an embodiment of thedisclosure. FIG. 5B is a view illustrating an embodiment to train anartificial intelligence assistant, according to an embodiment of thedisclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of anartificial intelligence assistant based on the application of userpreference information about the function execution (e.g.,identification of the intent of user speech) of the NLU module 220 ofFIG. 1D.

In this regard, referring to FIG. 5A, in operation 501, a user mayoperate a speech recognition service by executing an intelligence appinstalled in a user terminal. If the intelligence app is executed, or ifthe artificial intelligence assistant is activated depending on theexecution of the intelligence app, the user may perform speech includinga specific command or intent.

In operation 503, the activated artificial intelligence assistant mayrespond to the user speech. For example, the response corresponding tothe user speech may be generated based on function execution by at leastone of the ASR module 210 of FIG. 1D, the NLG module 250 of FIG. 1D, andthe TTS module 260 of FIG. 1D of the intelligence server 200 of FIG. 1D,and the intelligence agent (or the processor 150 of FIG. 1B) of the userterminal may output the response on a dialogue interface.

In operation 505, the intelligence agent may receive feedbackinformation about the execution result of a specific unit operation(e.g., final unit operation) according to a path rule, from the user. Inthe case where the feedback information of satisfaction attribute isprovided from the user, the intelligence agent may exclude thecalculation of the experience point of the artificial intelligenceassistant.

In the case where the feedback information of dissatisfaction attributeis provided from the user, in operation 507, the intelligence agent mayreceive, from the user, preference information associated with theexecution result of the specific unit operation and may transmit thefeedback information of the dissatisfaction attribute and the preferenceinformation to an intelligence server. To cope with the issue, the NLUmodule 220 of FIG. 1D may apply preference information of the user tothe intent on a matching rule consisting of a domain, intent, and aparameter (or slot). In this operation, the NLU module 220 may change aunit operation (or action) associated with the preference information ona path rule associated with the user speech intent before the preferenceinformation is applied. Hereinafter, an example associated withabove-described operation 505 and operation 507 will be described withreference to FIG. 5B.

Referring to a tenth dialogue interface 39 a of FIG. 5B, in the casewhere speech (e.g., “Show me the photo you captured yesterday”)including a specific command or intent is performed by the user, a userinput according to the speech may be transmitted to the ASR module ofthe intelligence server. In an embodiment, it is understood that theuser's specific command or intent included in the speech is view details(e.g., a state where a photo is enlarged in the screen area of anelectronic device) associated with a photo among at least one or morephotos captured yesterday. The ASR module may perform speech recognitionon a user input and, after converting the recognized user input into atext, the ASR module may transmit the text to the user terminal. The NLUmodule may analyze the speech recognition result (or converted text) todetermine a domain (e.g., photo), intent (e.g., show me a photo), and aparameter (e.g., yesterday). The NLU module may generate a path ruleincluding a first unit operation (e.g., execution of a galleryapplication), a second unit operation (e.g., date settings), a thirdunit operation (e.g., navigation of a gallery), and a fourth unitoperation (e.g., thumbnail display), based on the user speech intent ormay select the path rule on a database to transmit the selected pathrule to the user terminal. As such, the intelligence agent may outputtext information received from the ASR module on the tenth dialogueinterface 39 a and may output the execution screen of a specific unitoperation (e.g., the first unit operation (the execution of a galleryapplication)) to a fourth interface area 41 a, which is an area otherthan the tenth dialogue interface 39 a in the screen area of the userterminal.

Referring to an eleventh dialogue interface 39 b, the intelligence agentmay perform a final unit operation on a path rule and may transmitprocessing information thereof to the intelligence server. With regardto the processing information, the intelligence agent may output textinformation (e.g., I found five photos), which the NLG module transmits,and at least one object 43 supporting the user's feedback inputassociated with the final unit operation, on the eleventh dialogueinterface 39 b. In addition, the intelligence agent may output theexecution screen (e.g., a screen including fifth thumbnails) of a finalunit operation (e.g., the fourth unit operation (thumbnail display), ina fifth interface area 41 b of the screen area of the user terminal.

Referring to a twelfth dialogue interface 39 c, when the command orintent of the performed speech is a photo detail view, the user mayapply an input to a dissatisfaction object. In this case, the DM module240 of FIG. 1D may receive information about a user input from the userterminal and may transmit, to the user terminal, text information (e.g.,I can understand better if you let me know which part is the problem.I'll remember and I'll do better next time.) corresponding to the user'sdissatisfaction feedback. In this operation, the DM module 240 maytransmit list information about at least one word constituting the textconverted by the ASR module, to the user terminal. For example, it isunderstood that the list information includes yesterday tap 45,“captured” tap 47, “photo” tap 49, and “show me” tap 51. The user mayapply an input (e.g., touch) to “show me” tap 51 corresponding to viewdetails.

Referring to a thirteenth dialogue interface 39 d and a fourteenthdialogue interface 39 e, the DM module may transmit at least one groupof candidates' information to the user terminal, in response to theuser's touch input. As such, photo detail view tap 53, list view tap 55,or share tap 57 corresponding to the group of candidates' informationmay be displayed on the thirteenth dialogue interface 39 d. The user mayapply the touch input to a tab (e.g., photo detail view tap 53) closelyrelated to the speech intent. In this case, the NLU module may determinethat the photo detail view is user preference information, and may applythe preference information to intent on a matching rule composed of adomain, intent, and a parameter. As such, the NLU module may change theunit operation associated with the user's speech intent previouslyderived based on the application of the preference information of atleast one unit operation constituting the pass rule. For example, theNLU module may change the fourth unit operation from the thumbnaildisplay to photo detail view.

Returning to FIG. 5A, in operation 509, when the preference informationis applied to the user speech intent initially identified by the NLUmodule, the intelligence agent of a user terminal may assign a point(Pts) to the NLU module.

Pts _(n) −Pts _(n-1) +S _(intent)   Equation 5

Pts_(n): the final point of the NLU module

Pts_(n-1): the previous point of the NLU module

S_(intent): the unit score of user preference information applied to theintent on the matching rule

It is understood that Equation 5 is an equation associated with thescoring of the NLU module. The intelligence agent may add the unit scoreSintent of the user preference information to the point Pts_(n-1), whichhas been assigned to the NLU module, to assign the final point Pts_(n)to the NLU module.

Operation 511 may be the same as or correspond to operation 411described through FIG. 4A.

FIG. 5C is a flowchart illustrating a fifth embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an NLU module, according to an embodiment of thedisclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of anartificial intelligence assistant based on the application of userpreference information about the function execution (e.g., determinationof the parameter (or slot) of user speech) of the NLU module 220 of FIG.1D.

Referring to FIG. 5C, in operation 513, a user may control (e.g.,manipulating a hardware key on a user terminal or speaking a specifiedwakeup command) the execution of a speech recognition app to activatethe artificial intelligence assistant and may perform speech (e.g.,“call Professor Hong”) including a specific command or intent.

In operation 515, the intelligence agent may determine the clarity ofthe user input according to user speech. In this regard, the ASR module210 of FIG. 1D of the intelligence server 200 of FIG. 1D may performspeech recognition on the user input, and the NLU module 220 of FIG. 1Dmay analyze the speech recognition result (or the text converted by theASR module 210) to identify a domain (e.g., telephone), intent (e.g.,make a call), and a parameter (e.g., Professor Hong). The NLU module maygenerate a path rule including a first unit operation (e.g., executionof a contact application), a second unit operation (e.g., recipientsettings), and a third unit operation (e.g., Call connection), dependingon the identification of a user speech intent or may select the pathrule on a database to transmit the selected path rule to the userterminal. In an embodiment, in the case where a function (e.g., a callfunction) corresponding to user speech is performed on the user terminalafter the execution the at least one unit operation is completed, theintelligence agent may exclude the calculation of the experience pointof an artificial intelligence assistant.

Alternatively, in an embodiment, with regard to the execution of thesecond unit operation (e.g., recipient settings), pieces of information(e.g., Professor Gildong Hong and Professor Samsung Hong) correspondingto information about the parameter (e.g., Professor Hong) that the NLUmodule identifies may be present on a user terminal (or contactapplication). In this case, the intelligence agent may determine thatthe user input according to user speech is unclear.

If it is determined that the user input is unclear, in operation 517,the DM module 240 of FIG. 1D of the intelligence server may transmit agroup of candidates (e.g., Professor Gildong Hong tap and ProfessorSamsung Hong tap) of parameter (e.g., Professor Hong), that causes theuncertainty, to the user terminal. Alternatively, the DM module maytransmit additional information request (e.g., there are contacts ofProfessor Gildong Hong and Professor Samsung Hong. Who do you want tocall?) in the form of a text, to the user terminal.

In operation 519, a user may apply a touch input to one among the groupof candidates depending on the intent of the performed speech or maytouch one of Professor Gildong Hong or Professor Samsung Hong through aninput interface (e.g., SIP keyboard). Alternatively, in variousembodiments, the user may provide a user input associated with one ofProfessor Gildong Hong or Professor Samsung Hong, through speech.

In operation 521, the NLU module may determine that the candidateselected from the user or the professor (e.g., Professor Gildong Hong)corresponding to the user touch or the speech input is user preferenceinformation. The NLU module may apply the determined preferenceinformation to the initially identified parameter (e.g., Professor Hong)or may assign a priority to the initially identified parameter, withregard to the speech recognition result by the ASR module Afterwards,the NLU module may determine that the speech recognition resultincluding Professor Hong or the parameter associated with the convertedtext is Professor Gildong Hong.

In operation 523, the intelligence agent may assign the point Pts to theNLU module based on the application of the user preference informationabout the parameter that the NLU module identifies initially.

Pts _(n) =Pts _(n-1) +S _(param)   Equation 6

Pts_(n): the final point of the NLU module

Pts_(n-1): the previous point of the NLU module

S_(param): the unit score of user preference information applied to aparameter

It is understood that Equation 5 is another equation associated with thescoring of the NLU module. The intelligence agent may add the unit scoreS_(param) of the user preference information to the point Pts_(n-1),which has been assigned to the NLU module, to assign the final pointPts_(n) to the NLU module.

Operation 525 may be the same as or correspond to operation 411described through FIG. 4A.

FIG. 5D is a flowchart illustrating a sixth embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of an NLU module, according to an embodiment of thedisclosure.

FIG. 5E is a view illustrating an embodiment to train an artificialintelligence assistant, according to an embodiment of the disclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of anartificial intelligence assistant based on function response settingscorresponding to the function execution (e.g., identification of theintent of user speech) of the NLU module 220 of FIG. 1D.

In this regard, referring to FIG. 5D, in operation 527, the intelligenceagent (or the processor 150 of FIG. 1B) may output a sixth interfaceassociated with function settings of a speech recognition service, undercontrol of a user. For example, it is understood that the sixthinterface is at least part of the execution screen of an intelligenceapp.

In operation 529, the user may select one of at least one category onthe sixth interface and may set at least one function response (oraction) to be triggered by the speech associated with the correspondingcategory. In an embodiment, the category may be associated with at leastone of user location and user context, and each category may be referredto as a “recipe”. Hereinafter, an example associated with operation 529will be described with reference to FIG. 5E.

Referring to FIG. 5E, at least one category 63 (or recipe) associatedwith the location (e.g., Home, Work, or the like) and the context (e.g.,Going out, Get in car, Get off car, Sleeping, Walking up, or the like)of a user may be included on a sixth interface 61 a. In an embodiment,in the case where the user applies an input (e.g., touch) to the area ofa specific category (e.g., Home), the sixth interface 61 a may beswitched to a seventh interface 61 b supporting function responsesettings associated with the corresponding category.

At least one speech information 65 (e.g., “I am at home”, “Home mode”,“Coming home”, or the like) associated with the category selected from auser may be included on the seventh interface 61 b. For example, thespeech information 65 may include a word corresponding to the categoryor a word associated with the intent of the category. In addition,information 67 (e.g., Turn on Wi-Fi, Turn on Bluetooth, Turn on sound,or the like) about at least one function response (or action) may beincluded in one area of the seventh interface 61 b. In an embodiment, itis understood that the function response is an operation that is to beperformed by the intelligence agent after being triggered by theperformed speech, in the case where the user has performed speech on oneof the at least one speech information 65. In this regard, the functionresponse may be associated with the control of the function installed inthe user terminal or may be associated with the function control of atleast one external device (e.g., IoT device) that interacts with theuser terminal. The user may select and activate a specific functionresponse based on an input (e.g., touch); in this operation, thesequence of user inputs associated with the plurality of functionalresponses may act in sequence of a plurality of functional responses tobe performed by the intelligent agent.

Returning to FIG. 5D, in operation 531, the intelligence agent maytransmit, to the NLU module 220 of FIG. 1D of the intelligence server200 of FIG. 1D, speech information (e.g., “I am at home”, “Home mode”,“Coming home”, or the like) associated with a specific category (e.g.,Home) and information about the function response (or action) activatedwith respect to the speech information. Alternatively, in variousembodiments, at least one speech information associated with a specificcategory may be stored in the database 221 of FIG. 1D corresponding tothe NLU module, and the intelligence agent may transmit, to the NLUmodule, only the function response information activated by the user.The NLU module may map and store speech information about the specificcategory to function response information.

According to the above description, in the case where a user speechintent associated with the user input derived from the NLU modulecorresponds to the speech information designated with respect to thespecific category, the NLU module may specify (or determine) thefunction response, which is mapped to the speech information, as theuser speech intent. When the user speech intent is determined as thefunction response, the NLU module may generate or select a path ruleassociated with the execution of the function response. That is, in thecase where the user performs speech corresponding to the specificcategory, the intelligence agent may receive a path rule associated withthe function response mapped to the speech to perform at least one unitoperation, and thus perform the function response.

In operation 533, the intelligence agent may assign the point Pts to theNLU module based on function response setting associated with aspecified user speech.

Pts _(n) =Pts _(n-1) +S _(recipe)   Equation 7

Pts_(n): the final point of the NLU module

Pts_(n-1): the previous point of the NLU module

S_(recipe): the unit score of a function response set to a specificrecipe (or category)

It is understood that Equation 7 is yet another equation associated withthe scoring of the NLU module. The intelligence agent may add the unitscore S_(recipe) of the set function response to the point Pts_(n-1),which has been assigned to the NLU module, to assign the final pointPts_(n) to the NLU module. In an embodiment, in the case where there area plurality of function responses set (or activated) by the user, theunit score of each of the plurality of functions responses may be addedto the previous point Pts_(n-1) of the NLU module.

Operation 535 may be the same as or correspond to operation 411described through FIG. 4A.

FIG. 6A is a flowchart illustrating a seventh embodiment to calculate anexperience point of an artificial intelligence assistant based onfunction execution of a personal information server, according to anembodiment of the disclosure.

FIGS. 6B and 6C are views illustrating various embodiments to train anartificial intelligence assistant, according to various embodiments ofthe disclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of theartificial intelligence assistant based on user information in thedatabase established in the personal information server 300 of FIG. 1Aor the verification of usage information of the user terminal

In this regard, referring to FIG. 6A, in operation 601, the artificialintelligence assistant activated depending on the execution of anintelligence app may infer information (e.g., name, age, gender,address, occupation, health, anniversary, family, or the like)associated with the user, based on the operation pattern of the user'svoice recognition service, the operation history of the user's voicerecognition service, context information upon operating the user's voicerecognition service, or the like. In an embodiment, the inference mayinclude the verification of the user information or usage information ofthe user terminal pre-stored in the user terminal or the personalinformation server or the check of the clarity of the user informationor usage information of the user terminal to be stored in the personalinformation server.

In operation 603, the artificial intelligence assistant may request theuser's feedback associated with information to be verified or checkedthrough a query. As such, in operation 605, the user may respond to theauthenticity of the queried information. Hereinafter, an exampleassociated with operation 601 to operation 605 will be described withreference to FIGS. 6A and 6B.

Referring to FIG. 6B, the intelligence server may infer information(e.g., the user's current location) about the user based on information(e.g., user information (e g , name, age, gender, address, occupation,health, anniversary, family, or the like) or user terminal information(e.g., location information, communication information, applicationoperating information, or the like)) provided from the personalinformation server or the user terminal The NLG module 250 of FIG. 1D ofthe intelligence server may generate query information about theinferred information and may transmit the query information to the userterminal. As such, an eighth interface 69 including the query (e.g., Areyou home now?) by the artificial intelligence assistant may be output tothe user terminal. The user performs feedback speech (e.g., Yes, Thisplace is my home) as the response to the query, and the artificialintelligence assistant may perform a query (e.g., Maetan-dong,Yeongtong-gu, Suwon-si, Korea Right?) associated with the verificationof user information (e.g., address) provided from the personalinformation server. In the case where the user information is verifiedby the user response (e.g., Right), the intelligence server may confirmuser information provided from the personal information server or theuser terminal.

FIG. 6C is a view illustrating another example associated with operation601 to operation 605. And an example described in FIG. 6C may includethe verification or check process similar to the above-described that inFIG. 6B. For example, as illustrated in a ninth interface 71, in thecase where the user performs speech (e.g., Text Jack, Where are you?)including a specific command or intent, the intelligence server mayverify or check user preference information (e.g., the user preferenceassociated with persons, which have the same name, stored in the userterminal) associated with the user speech, based on user terminalinformation (e.g., application operating information, call information,communication information, or the like) provided from the personalinformation server. In this regard, the query (e.g., Jack, who youcalled yesterday?) of the artificial intelligence assistant associatedwith the verification or check may be displayed on the ninth interface71. The artificial intelligence assistant may perform verification orcheck query (e.g., Do you want to remember Jack as Jack Anderson?)associated with information (e.g., message recipient (Jack)) about theuser speech, based on the user's feedback speech (e.g., Right)associated with the query. In the case where the user performs feedbackspeech (e.g., Ok) on the verification or check query, the intelligenceserver may confirm user preference information provided from thepersonal information server or the user terminal.

Returning to FIG. 6A, in operation 607, in the case where the queryinformation of the artificial intelligence assistant is verified orchecked depending on the user's response, the intelligence agent mayassign the point Pts of a specified value to the personal informationserver based on the interaction between the artificial intelligenceassistant and the user with respect to the verification or checkprocess.

Operation 609 may be the same as or correspond to operation 411described through FIG. 4A.

FIGS. 7A to 7D are views illustrating various embodiments to calculatean experience point of an artificial intelligence assistant based onexecution of user activity, according to an embodiment of thedisclosure.

In an embodiment, the intelligence agent 151 of FIG. 1B of the userterminal 100 of FIG. 1B may calculate the experience point of theartificial intelligence assistant based on a user's activity performedon a user terminal or an intelligence app.

In this regard, referring to FIG. 7A, the user may apply a touch input89 to a tenth interface 87 a associated with the execution screen of theintelligence app, for example, the training or experience category (ormenu) of the artificial intelligence assistant. In this case, the tenthinterface 87 a may be switched to an eleventh interface 87 b includingat least one activity information 91 that trains or grows the artificialintelligence assistant. The user may select an activity (e.g., useraddress information providing activity) to interact with the artificialintelligence assistant. For example, the user may perform interaction(e.g., query and response), which is associated with the correspondingactivity, with the artificial intelligence assistant on a dialogueinterface 93 output in response to the selection of the activity. In anembodiment, in the case where the activity selected from the user iscompleted, an icon 95 a of the completed activity may be displayed(e.g., color display or flash blink) as a specified display on the tenthinterface 87 a and the eleventh interface 87 b. In this case, withregard to the execution of the corresponding activity, the intelligenceagent may assign a point of a pre-specified value as a user activitypoint and may calculate the experience point of the artificialintelligence assistant by summing the point of a pre-specified value andany other point (e.g., the point of the ASR module 210 of FIG. 1D, thepoint of the NLU module 220 of FIG. 1D, the point of the personalinformation server 300 of FIG. 1A, or the like).

Alternatively, referring to FIG. 7B, at least one usage guideinformation 99 associated with the operation of the speech recognitionservice or the intelligence app may be displayed on a twelfth interface97 associated with the execution screen of the intelligence app, asanother example of the training or experience of the artificialintelligence assistant. The user may select any usage guide of the usageguide information 99 to perform the corresponding activity. In thiscase, the intelligence agent may assign the point of a pre-specifiedvalue to each usage guide information 99 as a user activity point.

Referring to FIG. 7C, in an embodiment, the user may share informationwith acquaintances, such as family members, friends, or the like basedon the operation of the speech recognition service. In this regard, theintelligence server 200 of FIG. 1D may identify the acquaintancesassociated with the user based on user information (e.g., familyinformation or the like) and usage information (e.g., application(contacts) information, social media service account information, or thelike) of the user terminal, which are provided from the user terminal orthe personal information server. Alternatively, the acquaintance may bepre-assigned by the user control or recommended by the user. In anembodiment, in an operation of interacting with the user through adialogue interface 101, in the case where there is no informationaccompanying the processing of the user's speech, the artificialintelligence assistant may query whether information share is requestedfrom the identified acquaintance. The generation of the query may beimplemented by the function execution of the NLG module 250 of FIG. 1Dof an intelligence server. Alternatively, as in that described above,the artificial intelligence assistant may query whether information isprovided to the acquaintance. In the case where the information isshared from the user terminal to an external device (e.g., a terminalowned by the acquaintance) or in the case where the information isshared from the external device to the user terminal, the intelligenceagent may assign the specified point as a user activity point.

Referring to FIG. 7D, in an embodiment, the intelligence server 200 maycollect pieces of activity information performed by the correspondingexternal device, from at least one external device 105, 107 or 109(e.g., a terminal owned by the acquaintance associated with the user)and may analyze the activity information to identify an activity towhich a high point is assigned, or an activity that is not performed onthe user terminal 100. In this regard, the artificial intelligenceassistant may perform a query, which recommends or suggests theexecution of an activity of a high point that the acquaintance performsor an activity that the user does not perform, to the user based on adialogue interface 103. In the case where the recommendation orsuggestion is accepted by the user's feedback speech, the intelligenceagent may assign the specified point as a user activity point.

FIGS. 8A to 8G are views illustrating various embodiments to use anexperience point of an artificial intelligence assistant, according toan embodiment of the disclosure.

Referring to FIG. 8A, the user terminal 100 may receive a messageaccording to the speech of the external device user, from an externaldevice (e.g., a terminal operating a speech recognition service or anintelligence app). As such, a message application may be executed on theuser terminal 100, and the message information transmitted by theexternal device may be displayed on an execution screen 111 of themessage application. At this time, information about the transmissionmeans (or path) of the message may be included in the messageinformation. For example, when the message is transmitted based on theoperation of a speech recognition service (or the operation of anintelligence app) of a user of the external device, information (e.g.,the experience point of the artificial intelligence assistant, the iconof the artificial intelligence assistant, the level information of theartificial intelligence assistant, the visual graph associated with theexperience point of the artificial intelligence assistant, or the like)about the artificial intelligence assistant of the external device maybe included in a message information displayed on the execution screen111. As such, the user of the user terminal 100 may determine the degreeof trust in the received message. For example, the user of the userterminal 100 may estimate the accuracy of speech recognition performedby the external device based on the experience point of the artificialintelligence assistant of the external device included in the messageinformation to estimate a typo error 113 (e.g., in the case where a userof the external device utters “about” but the external device recognizes“about” as “information” because speech recognition is performedunclearly, and thus a message including a typo is transmitted to theuser terminal 100) in the message information or information that is notcapable of being recognized. In an embodiment, in the case whereinformation about the artificial intelligence assistant in the messageis selected (or touched), detailed information (e.g., trust percentageinformation or error probability information of speech recognition)about the artificial intelligence assistant may be displayed.

Referring to FIG. 8B, the intelligence server 200 of FIG. 1D maystepwisely support the function of the speech recognition service to theuser terminal 100, based on the experience point of the artificialintelligence assistant on the user terminal 100. In this regard, it isunderstood that in the case of the slight experience point of theartificial intelligence assistant, the function execution of the element(e.g., the ASR module 210 of FIG. 1D, the NLU module 220 of FIG. 1D, orthe like) of the intelligence server 200 is not clear or the update of amodel associated with the element is not complete. In this case, thespeech recognition associated with the user's speech input or anoperation of generating a path rule may be performed unclearly or anerror may occur during the performance. As such, the intelligence server200 may provide a stepwise service function depending on the experiencepoint of the artificial intelligence assistant of the user terminal 100.In the case where the experience point of the artificial intelligenceassistant on the user terminal 100 exceeds a specified critical value,the intelligence server 200 may refine the function support of thespeech recognition service corresponding to the corresponding criticalvalue with respect to the user terminal 100. The NLG module 250 of theintelligence server 200 may allow the artificial intelligence assistantto provide information about the refined function (e.g., a function tosend a message by using speech) through a dialogue interface 115.Alternatively, the intelligence server 200 may refine the servicefunction support associated with the function execution of the elementcorresponding to a high point, with reference to a plurality of points(e.g., the point of the ASR module 210 of FIG. 1D, the point of the NLUmodule 220 of FIG. 1D, the point of the personal information server 300of FIG. 1A, and the like) contributing to the calculation of theexperience point of the artificial intelligence assistant. In anembodiment, in the case where a touch input is applied to the dialogueinterface 115 (or in the case where a touch input is applied to an areaof information about the refined function), a thirteenth interface 117in which the service function provided to the specific experience point(or of which the support is refined) is listed may be output. In variousembodiments, the user may release the operating restriction with respectto specific functions on the thirteenth interface 117 regardless of thedegree of the current experience point of the artificial intelligenceassistant and may request the intelligence server 200 to support thereleased service function.

Referring to FIG. 8C, the intelligence server 200 may support, forexample, the update of an icon 119 of the artificial intelligenceassistant displayed on a dialogue interface, based on the experiencepoint the artificial intelligence assistant of the user terminal 100.For example, the intelligence server 200 may provide icons 121 and 123of various themes or may provide an icon 125 of various races orcharacters, depending on the experience point. Alternatively, theintelligence server 200 may support an icon 127 of a scenario in whichan animal, character, person, or the like grows in response to the riseof the experience point.

Referring to FIG. 8D, level information corresponding to the experiencepoint of the artificial intelligence assistant may be displayed on afourteenth interface 129 associated with the execution screen of anintelligence app. In an embodiment, in the case where the experiencepoint reaches the specific level, coupons, merchandise, vouchers, or thelike provided from an external device (e.g., shopping mall server,grocery server, application market server, or the like) associated withor in cooperation with the integrated intelligent system 10 of FIG. 1Amay be provided (e.g., delivered or downloaded).

Referring to FIG. 8E, the user may set a voice corresponding to theexperience point of the artificial intelligence assistant on the userterminal 100, on the fifteenth interface 131 associated with theintelligence app. It is understood that the voice is the acousticcharacteristic associated with the response (e.g., accent, tone, speechspeed, or the like) in the case where the response of the artificialintelligence assistant is output through the speaker 130 of FIG. 1B onthe user terminal 100. In this regard, voice information correspondingto a specific experience point may be displayed on the fifteenthinterface 131. For example, a voice list to be grown depending on therise of the experience point may be displayed on the fifteenth interface131, and the user may select and activate a voice corresponding to theexperience point of the artificial intelligence assistant calculated asneeded. In various embodiments, in an operation of selecting a voice,the user may allow the voice to be selectively applied to only wordsthat frequently appear or are expressed during the response of theartificial intelligence assistant.

Referring to FIG. 8F, information (e.g., an experience point value, apath where the experience point is obtained, statistical informationabout the experience point, or the like) about the experience point ofthe artificial intelligence assistant may be displayed on a sixteenthinterface 133 associated with the execution screen of an intelligenceapp. In an embodiment, the sixteenth interface 133 may include an object135 (e.g., boasting) that is capable of transmitting or sharing theabove-described information onto a social media service installed in theuser terminal 100. In the case where the user's touch input is appliedto the object 135, at least one experience point information included inthe sixteenth interface may be updated in a user feed 136 on the socialmedia service.

Referring to FIG. 8G, in the case where the experience point of theartificial intelligence assistant on the user terminal 100 exceeds aspecified value, the intelligence server 200 of FIG. 1D may providevisual effect associated with the artificial intelligence assistant. Forexample, in the case where the intelligence app is executed or in thecase where the artificial intelligence assistant is activated, theintelligence server 200 may provide the icon of the artificialintelligence assistant with visual effect on a home screen or a lockscreen of the user terminal 100. In this regard, in the case where usermanipulation is applied to a hardware key 112 of FIG. 1C on the userterminal 100 in the sleep state of the user terminal 100 or in the casewhere the user performs specified wakeup command speech, the icon of theartificial intelligence assistant and a particle object of a specifiedshape may be displayed on a first lock screen 137 a of the user terminal100. Afterwards, as specified time goes on, the particle object may flowand form gradually a specified text as illustrated in a second lockscreen 137 b. If the flow of the particle object is completed, a textindicating specific information may be displayed on the third lockscreen 137 c.

FIG. 9A is a block diagram illustrating an architecture associated withsome elements in an integrated intelligent system, according to anembodiment of the disclosure. FIGS. 9B through 9K are views illustratingvarious processes associated with some elements of an architecture andan interface of a relevant user terminal, according to variousembodiments of the disclosure.

Referring to FIGS. 9A through FIG. 9K, additional various embodiments inaddition to the above-described various embodiments will be describedwith regard to the calculation of the experience point of an artificialintelligence assistant. In addition, the function execution of elementsimplementing the additional various embodiments and relevant screeninterfaces will be described. In various embodiments, it is understoodthat an intelligence server described below is the above-describedintelligence server 200 of FIG. 1D or a server (e.g., level informationmanagement server) implemented separately for the purpose of managingthe experience point or the level information of the artificialintelligence assistant. For convenience of description, hereinafter, theintelligence server may be referred to as the above-described“intelligence server 200” of FIG. 1D.

Referring to FIGS. 9A and 9B, in association with the operation ofspeech recognition service of the user, the intelligence agent 151 maymanage the specific activity of the user as usage information and mayreceive a point used to calculate the experience point of the artificialintelligence assistant, from the intelligence server 200 based on theusage information. The specific activity may include feedback provisionof the user's satisfaction or dissatisfaction attribute with respect tothe response of the artificial intelligence assistant provided dependingon user speech. Alternatively, the specific activity may includefunction operation control (e.g., dialing, sending a message, changingsystem settings of an intelligence app, or the like) associated with atleast one app mounted or installed in the user terminal 100 of FIG. 1Bby using the artificial intelligence assistant. According to anembodiment, the function operation control associated with the app mayaccompany the execution or processing of a path rule provided from theintelligence server 200 to the user terminal 100 depending on the userspeech. In other words, the specific activity may include a series ofprocesses in which the user terminal 100 receives a path rulecorresponding to the user speech to perform or process the path ruledepending on user speech (e.g., please make a call, please send amessage, please change settings, or the like) including a specificintent or command. At this time, the intelligence agent 151 may manageinformation (e.g., path rule ID) about the path rule received inresponse to the execution of specific activity of the user. In the casewhere the above-described specific activity of the user occurs, theintelligence agent 151 may transmit usage information to theintelligence server 200.

In an embodiment, the intelligence server 200 may establish thepredetermined point assignment value as an index with respect to eachspecific activity. In the case where the intelligence server 200receives the usage information from the intelligence agent 151, theintelligence server 200 may track the activity corresponding to theusage information on the index to identify the corresponding pointassignment value. The intelligence server 200 may transmit informationabout the identified point assignment value to the intelligence agent151, and the intelligence agent 151 may refer to the receivedinformation to calculate the experience point of the artificialintelligence assistant. Alternatively, in various embodiments, theintelligence server 200 may receive level information from theintelligence agent 151 and may apply the point assignment value to thelevel information to transmit the refined level information to theintelligence agent 151.

As described above, the calculation of the experience point of theartificial intelligence assistant may be performed by a scoring managermodule included in the intelligence server 200. In this regard,referring to FIGS. 9A and 9C, the intelligence server 200 may managespeech recognition service subscription information of the user and maymap and manage the subscribed user information to the experience pointof the artificial intelligence assistant or level informationcorresponding to the experience point. In an embodiment, in associationwith the experience point or level information of the artificialintelligence assistant to be displayed on the screen interface of anintelligence app, in the case where the intelligence app is executed,the intelligence agent 151 may make a request for the updated experiencepoint or level information to the intelligence server 200. Theintelligence server 200 may transmit, to the intelligence agent 151, theupdated experience point or level information as the response to therequest, and the intelligence agent 151 may display the receivedexperience point or level information on the screen interface.

In an embodiment, the intelligence server 200 may update the experiencepoint or level information of the artificial intelligence assistant inreal time and may provide information, which is changed depending on theupdate, to the intelligence agent 151. For example, in the case wherethe level grade corresponding to the experience point is changed becausethe experience point of the artificial intelligence assistant increase,the intelligence server 200 may transmit the changed level gradeinformation to the intelligence agent 151. In the case where theexisting level information is changed based on the level gradeinformation, the intelligence agent 151 may output a notification (e.g.,message, sound effect, or the like) thereof

According to an embodiment, the intelligence server 200 may transmit theexperience point or level information of the artificial intelligenceassistant operated by the user terminal 100, to a speech recognitionservice portal 500 implemented through a separate server. The speechrecognition service portal 500 may collectively manage at least one usersubscribed in the speech recognition service and may collectively managethe experience point or level information of the artificial intelligenceassistant operated by the user terminal of each user. In an embodiment,the intelligence server 200 may request the specific reward or benefits(hereinafter collectively referred to as a “reward”) corresponding tothe corresponding experience point or level information together withthe transmission of the experience point or level information to thespeech recognition service portal 500. In this regard, the speechrecognition service portal 500 may establish a reward platform with afirst external device (e.g., payment service server) associated with thespeech recognition service and may transmit, to the first externaldevice, the reward information requested from the intelligence server200 and user information (e.g., speech recognition service subscriptioninformation, user subscription ID, or the like) associated with thereward. In various embodiments, the reward may include the accumulationof a point associated with a service operated by the first externaldevice. The first external device may process (e.g., earn points) rewardinformation received from the speech recognition service portal 500 totransmit the processing result to the speech recognition service portal500 and may manage information about the processing history. As such,the speech recognition service portal 500 may manage the reward historyof the user terminal corresponding to the reward requested by theintelligence server 200, based on the processing result received fromthe first external device. In various embodiments, the intelligenceserver 200 may exclude the reward request to the speech recognitionservice portal 500, may make a request for the reward to the firstexternal device directly, and may receive the sharing of processinginformation about the reward request from the first external device tomanage (e.g., manage the reward history) the processing information.Moreover, the intelligence server 200 may transmit the reward history tothe intelligence agent 151; as such, the intelligence agent 151 mayoutput the reward history through the interface 16 of FIG. 2C describedthrough FIG. 2C. The above-described reward request and processing maybe based on the assumption that the user associated with the rewardsubscribes the service operated by the first external device, and in thecase where the intelligence agent 151 receives the reward history fromthe intelligence server 200, the intelligence agent 151 may access thefirst external device to determine whether the user subscribes theservice.

Referring to FIG. 9A and 9D, the speech recognition service portal 500may manage at least one promotion associated with the operation of thespeech recognition service. For example, the speech recognition serviceportal 500 may establish the promotion platform with the second externaldevice (e.g., an affiliate product server or the like) associated withthe speech recognition service. In an embodiment, the manager (oroperator) of the second external device may access the speechrecognition service portal 500 to register information about thespecific promotion. For example, the second external device manager mayregister promotion identification information (e.g., ID or promotionname allocated to the promotion, or the like), a promotion operationperiod, content information accompanying the operation of the promotion,information about promotion terms and conditions, or the like, in thespeech recognition service portal 500. The promotion informationregistered in the speech recognition service portal 500 may betransmitted to the user terminal 100 and/or the intelligence server 200,and the intelligence agent 151 may generate an event bulletin board onthe screen interface of an intelligence app to display the promotioninformation. In various embodiments, the second external device managermay exclude the registration of the promotion information to the speechrecognition service portal 500, and may operate a separate portal (e.g.,administration portal) to provide the intelligence agent with thepromotion information based on the separate portal.

In the case where the intelligence agent 151 may receive new promotioninformation from the speech recognition service portal 500 or in thecase where the promotion information is posted on the event bulletinboard, the intelligence agent 151 may output a notification (e.g., apush message or the like) thereof on the user terminal 100 to provide anotification of the promotion. The user may verify the promotion on theevent bulletin board and may enter the agreement information about theterms and conditions of the promotion in which the user wants toparticipate, in response to the notification. The intelligence agent 151may transmit the agreement information about the terms and conditions tothe speech recognition service portal 500, and the speech recognitionservice portal 500 may map and manage the received agreement informationabout the terms and conditions to the corresponding promotion.

According to an embodiment, the promotion may include various missionsfor activating the operation of the speech recognition service. Forexample, the second external device manager may constitute a mission ofthe execution of an intelligence app, a first user activity associatedwith the above-described usage information, a second user activity(e.g., finding a hidden function of the artificial intelligenceassistant, generating and using a macro, or the like) associated withthe function of the artificial intelligence assistant, a third useractivity (e.g., launching a plurality of apps based on multitaskingthrough user speech) using the artificial intelligence assistant, or thelike, as a promotion

FIG. 9E illustrates a series of processes according to theabove-mentioned promotion participation. In FIG. 9E, various embodimentsof the promotion participation may be represented by branching in aspecific operation (e.g., a stamp acquisition operation) in the process.

Referring to FIG. 9E, in the case where the promotion is announced onthe user terminal 100, the user may access the event bulletin board andmay verify the stamp information (e.g., stamp acquisition status) byagreeing with terms and conditions of the promotion in which the userdesires to participate. In an embodiment, if the user will perform atleast one mission consisting of promotions, the stamp may be providedfrom the second external device. In an embodiment, whenever obtainingthe stamp, the intelligence agent 151 may receive a predetermined pointfrom the second external device. Alternatively, in another embodiment,the intelligence agent 151 may receive the predetermined point from thesecond external device only when stamps of the specified number areauthenticated. The intelligence agent 151 may refer to the receivedpredetermined point to calculate the experience point of the artificialintelligence assistant.

FIG. 9F is a view illustrating various interfaces of an intelligence appassociated with promotion participation. Referring to FIG. 9F, aspecific interface 139 a (e.g., the home screen of an intelligence app),which is output depending on the execution of an intelligence app, maydisplay a menu associated with the operation of a speech recognitionservice. In an embodiment, the menu may include various categories, andthe specific category may support interworking with an event bulletinboard 139 b in response to a user's touch input. In the case where thespecific promotion is selected on the event bulletin board 139 b by theuser, the event bulletin board 139 b may be switched to an interface 139c including information about the selected promotion and a promotionparticipation tap. Furthermore, if the user's touch input is applied tothe promotion participation tap, an interface 139 d for enteringinformation about terms and conditions of the corresponding promotionand agreement information of the user about the terms and conditions maybe output.

Referring to FIGS. 9A and 9G, the intelligence agent 151 may receivevariation suggestion associated with the operation of the speechrecognition service from the user and may manage the variationsuggestion. For example, a touch input is applied to a first button(e.g., variation suggestion button) on a specific screen interface(e.g., variation suggestion interface) of an intelligence app by theuser; afterwards, in the case where a second button (e.g., variationsuggestion submitting button) is manipulated by performing the variationsuggestion of a user, the intelligence agent 151 may receive thevariation suggestion. In an embodiment, for example, the variationsuggestion may include another response suggestion of the userassociated with the response of an artificial intelligence assistantaccording to user speech. Alternatively, in the case where a path ruleis provided from the intelligence server 200 depending on the occurrenceof user speech associated with function operation control of a specificapp and then the function operation control of the specific app isprocessed, the variation suggestion may include suggestion (e.g., “senda message” is changed to “fly a message”, or the like) for at leastpartly changing the aspect of the user speech. In the case where theabove-described variation suggestion is provided from the user, theintelligence agent 151 may determine the validity (e.g., whether thevariation suggestion is suggestion of a specified expression) of theprovided variation suggestion to transmit (or stores the variationsuggestion in a user suggestion management database) the variationsuggestion to a variation suggestion management server 600 implemented(or implemented to include the intelligence server 200 or implementedwith the intelligence server 200 as it is) with a separate externaldevice. In this operation, the intelligence agent 151 may receiveacceptance information of the variation suggestion from the variationsuggestion management server 600 and may output a notification (e.g.,push message) including the acceptance information. In variousembodiments, in the case where it is determined that the variationsuggestion is invalid, the intelligence agent 151 may output aninterface for performing or entering the variation suggestion again.

In an embodiment, the occurrence of the variation suggestion mayaccompany the payment of predetermined point, and in the case where itis determined that the variation suggestion is valid, the intelligenceagent 151 may make a request for the point according to the occurrenceof the variation suggestion to the intelligence server 200. Theintelligence server 200 may apply the predetermined point (e.g., 10points) to the calculation of the experience point of the artificialintelligence assistant or the determination of the level information ofthe artificial intelligence assistant depending on the request. Thevariation suggestion management server 600 may map and store theprovided variation suggestion onto the corresponding user ID and mayprovide information thereof to the intelligence agent 151.Alternatively, the variation suggestion management server 600 maytransmit the provided variation suggestion to the speech recognitionservice portal 500. In an embodiment, the manager of the speechrecognition service portal 500 may determine the priority based on theefficiency for the received at least one variation suggestion and mayadopt the best (or top priority) variation suggestion to manage the bestvariation suggestion by using the list. The speech recognition serviceportal 500 may transmit predetermined point (e.g., 100 points) paymentinformation about the adopted variation suggestion, to the intelligenceserver 200, and the intelligence server 200 may apply the point (e.g.,100 points) according to the adoption of the variation suggestion to thecalculation of the experience point or the determination of the levelinformation of the artificial intelligence assistant. In an embodiment,the speech recognition service portal 500 may provide information aboutthe adopted variation suggestion to a training tool associated with theartificial intelligence assistant. Alternatively, the training tool mayrequest the speech recognition service portal 500 to access a listincluding the adopted at least one variation suggestion.

FIGS. 9H and 9I are views illustrating interfaces of an intelligence appin which the above-described variation suggestion is performed by auser. Referring to FIG. 9H, feedback provision history information ofthe user's satisfaction or dissatisfaction attribute with respect tohistory information (e.g., dialogue history) about interaction betweenthe user and the artificial intelligence assistant and the response ofthe artificial intelligence assistant according to the interaction maybe displayed on a first interface 145 a of an intelligence app. In anembodiment, in the case where a user touch input is applied to theinteraction history information, the first interface 145 a may beswitched to a second interface 145 b for suggesting another responsewith respect to the response of the artificial intelligence assistantaccording to the corresponding interaction. Referring to FIG. 9I, theuser may perform variation suggestion for at least partly changing theaspect of the specific user speech on a third interface 147 of theintelligence app. For example, the user may enter the aspect of the userspeech to be changed through a software input panel (SIP) keyboardprovided to at least one area of the third interface 147, and maytransmit (e.g., submit) the aspect of the user speech to the variationsuggestion management server 600.

FIG. 9J is a view illustrating an interface output by the user terminal100, in association with adoption of the variation suggestion. Referringto FIG. 9J, in the case where the specific variation suggestion isadopted by the speech recognition service portal 500, the user terminal100 may receive information about the adoption and may output anotification (e.g., push message) thereof For example, the user terminal100 may display the push message in at least one area of a homeinterface, a background interface, or an execution interface 148 of aspecific app and may display detailed information of the variationsuggestion adopted in response to a user's manipulation (e.g., drag)applied to the push message.

In an embodiment, the intelligence server 200 may refine the experiencepoint of a voice recognition service depending on the execution of theuser's variation suggestion. At this time, the experience point may berefined within a specified limit (e.g., 200 points per day), and in thecase where the intelligence server 200 monitors the refinement of theexperience point and the limit is exceeded, the intelligence server 200may transmit information thereof to the user terminal 100. However, thepoint (e.g., 100 points) received by adopting the above-describedsuggestion change may be excluded from the limit In this regard,referring to FIG. 9K, the user terminal 100 may output informationindicating the limit excess of the experience point through a separatemessage or the screen interface 149 of an intelligence app. As such, itis understood that the limit refine configuration of the experiencepoint is a part of the policy to prevent the operation of illegal speechrecognition services.

According to various embodiments, an electronic system may include auser device including a display, a microphone, a speaker, acommunication circuit, a first memory, and a first processorelectrically connected with the display, the microphone, the speaker,the communication circuit, and the first memory, and a first serverincluding a communication interface, a second memory, and a secondprocessor electrically connected with the communication interface, andthe second memory.

According to various embodiments, the second memory may include anautomatic speech recognition (ASR) module and a natural languageunderstanding (NLU) module.

According to various embodiments, the first memory may storeinstructions that, when executed by the first processor, causes thefirst processor to receive a voice input of a user through themicrophone and to transmit data associated with the voice input to thesecond processor through the communication circuit.

According to various embodiments, the second memory may storeinstructions that, when executed by the second processor, causes thesecond processor to process the data by using at least one of the ASRmodule or the NLU module, to calculate at least one numerical valueassociated with at least one of the ASR module or the NLU module byusing the data, and to provide the at least one numerical value to thefirst processor.

According to various embodiments, the first memory may further storeinstructions that, when executed by the first processor, causes thefirst processor to provide information about the at least one numericalvalue on a user interface.

According to various embodiments, an electronic system may include afirst server including a communication interface, a first memory, and afirst processor electrically connected with the communication interface,and the first memory.

According to various embodiments, the first memory may include anautomatic speech recognition (ASR) module and a natural languageunderstanding (NLU) module.

According to various embodiments, the first memory may storeinstructions that, when executed by the first processor, causes thefirst processor to receive data associated with a voice input throughthe communication interface, to process the data by using at least oneof the ASR module or the NLU module, to calculate at least one numericalvalue associated with at least one of the ASR module or the NLU modulebased on the processing of the data, and to transmit the at least onenumerical value to a specified external device through the communicationinterface.

According to various embodiments, an electronic device supporting aspeech recognition service may include a communication modulecommunicating with at least one external device, a microphone receivinga voice input according to user speech, a memory storing informationabout an operation of the speech recognition service, a displayoutputting a screen associated with the operation of the speechrecognition service, and a processor electrically connected to thecommunication module, the microphone, the memory, and the display.

According to various embodiments, the processor may be configured tocalculate a specified numerical value associated with the operation ofthe speech recognition service, to transmit information about thenumerical value to a first external device processing the voice input,and to transmit a request for a function, which corresponds to thecalculated numerical value of at least one function associated with thespeech recognition service stepwisely provided from the first externaldevice depending on a numerical value, to the first external device torefine a function of the speech recognition service supported by theelectronic device.

According to various embodiments, the processor may be furtherconfigured to assign a point to at least one of an automatic speechrecognition (ASR) module or a natural language understanding (NLU)module included in the first external device in association withfunction execution of the first external device and to calculate thenumerical value based on collection of the assigned point.

According to various embodiments, the processor may be furtherconfigured to collect at least one voice input information on whichspeaker-dependent speech recognition is performed by the ASR module, toaccumulate and calculate a user speech time corresponding to thecollected at least one voice input information, and to assign the pointto the ASR module based on an accumulated amount of the user speechtime.

According to various embodiments, the processor may be furtherconfigured to assign the point to the ASR module based on generation ofa speaker-dependent recognition model with respect to wakeup-commandspeech associated with the operation of the speech recognition service.

According to various embodiments, the processor may be furtherconfigured, in association with speech recognition execution of the ASRmodule with respect to the voice input, if an error of the speechrecognition result is revised, to assign the point to the ASR modulebased on a speech recognition model update of the ASR module performedin response to the revision of the error.

According to various embodiments, the processor may be furtherconfigured, in association with derivation execution of user speechintent of the NLU module with respect to the voice input, if userpreference information provided from a user is applied to at least oneof a domain, intent, or a parameter associated with the voice inputobtained by the NLU module, to assign the point to the NLU module basedon application of the user preference information.

According to various embodiments, the processor may be furtherconfigured, in association with derivation execution of user speechintent of the NLU module with respect to the voice input, if at leastone function response associated with function control of the electronicdevice or function control of a second external device interacting withthe electronic device is set with respect to a specific user speechintent to be derived by the NLU module, to assign the point to the NLUmodule based on the setting of the at least one function response.

According to various embodiments, the processor may be furtherconfigured to assign a point to a third external device receiving andstoring at least one of information about the electronic device orinformation about a user of the electronic device from the electronicdevice and to calculate the numerical value based on the assigned point.

According to various embodiments, the processor may be furtherconfigured to receive and output query information about verification orcheck of the at least one information stored in the third externaldevice, from the first external device and, if the at least oneinformation is verified or checked by a user feedback associated withthe query information, to assign the point to the third external devicebased on the verification or the check of the at least one information.

According to various embodiments, a speech recognition service operatingmethod of an electronic device may include receiving a voice inputaccording to user speech, calculating a specified numerical value inassociation with an operation of the speech recognition service,transmitting at least one of information about the voice input orinformation about the numerical value to a first external deviceprocessing the voice input, transmitting a request for a function, whichcorresponds to the calculated numerical value, of at least one functionassociated with the speech recognition service stepwisely provided fromthe first external device depending on a numerical value, to the firstexternal device, and receiving the function corresponding to thecalculated numerical value from the first external device to refine afunction of the speech recognition service.

According to various embodiments, the calculating may include assigninga point to at least one of an automatic speech recognition (ASR) moduleor a natural language understanding (NLU) module included in the firstexternal device in association with function execution of the firstexternal device and calculating the numerical value based on collectionof the assigned point.

According to various embodiments, the assigning may include collectingat least one voice input information on which speaker-dependent speechrecognition is performed by the ASR module, accumulating and calculatinga user speech time corresponding to the collected at least one voiceinput information, and assigning the point to the ASR module based on anaccumulated amount of the user speech time.

According to various embodiments, the assigning may include assigningthe point to the ASR module based on generation of a speaker-dependentrecognition model with respect to wakeup-command speech associated withthe operation of the speech recognition service.

According to various embodiments, the assigning may include, if an errorof a speech recognition result of the ASR module with respect to thevoice input is revised, assigning the point to the ASR module based on aspeech recognition model update of the ASR module performed in responseto the revision of the error.

According to various embodiments, the assigning may include, if userpreference information provided from a user is applied to at least oneof a domain, intent, or a parameter associated with the voice inputobtained by the NLU module in an operation of deriving user speechintent of the NLU module with respect to the voice input, assigning thepoint to the NLU module based on application of the user preferenceinformation.

According to various embodiments, the assigning may include, if at leastone function response associated with function control of the electronicdevice or function control of a second external device interacting withthe electronic device is set with respect to a specific user speechintent to be derived by the NLU module deriving user speech intentassociated with the voice input, assigning the point to the NLU modulebased on the setting of the at least one function response.

According to various embodiments, the calculating may include assigninga point to a third external device storing at least one of informationabout the electronic device or information about a user of theelectronic device and calculating the numerical value based on theassigned point.

According to various embodiments, the assigning may include receivingand outputting query information about verification or check of at leastone information stored in the third external device, from the firstexternal device, verifying and checking the at least one information bya user feedback associated with the query information, and assigning thepoint to the third external device based on the verification or thecheck of the at least one information.

FIG. 10 illustrates an electronic device (or user terminal) in a networkenvironment, according to an embodiment of the disclosure.

Referring to FIG. 10, under the network environment 1000, the electronicdevice 1001 (e.g., the user terminal 100 of FIG. 1B) may communicatewith an electronic device 1002 through a first network 1098 (e.g. awireless local area network such as Bluetooth or infrared dataassociation (IrDA)) or may communicate with an electronic device 1004 ora server 1008 through a second network 1099 (e.g., a wireless wide areanetwork such as a cellular network). According to an embodiment, theelectronic device 1001 may communicate with the electronic device 1004through the server 1008.

According to an embodiment, the electronic device 1001 may include a bus1010, a processor 1020 (e.g., the processor 150 of FIG. 1B) a memory1030, an input device 1050 (e.g., a micro-phone or a mouse), a display1060, an audio module 1070, a sensor module 1076, an interface 1077, ahaptic module 1079, a camera module 1080, a power management module1088, a battery 1089, a communication module 1090, and a subscriberidentification module 1096. According to an embodiment, the electronicdevice 1001 may not include at least one (e.g., the display 1060 or thecamera module 1080) of the above-described elements or may furtherinclude other element(s).

For example, the bus 1010 may interconnect the above-described elements1020 to 1090 and may include a circuit for conveying signals (e.g., acontrol message or data) between the above-described elements. Theprocessor 1020 may include one or more of a central processing unit(CPU), an application processor (AP), a graphic processing unit (GPU),an image signal processor (ISP) of a camera or a communication processor(CP). According to an embodiment, the processor 1020 may be implementedwith a system on chip (SoC) or a system in package (SiP). For example,the processor 1020 may drive an operating system (OS) or an applicationto control at least one of another element (e.g., hardware or softwareelement) connected to the processor 1020 and may process and computevarious data. The processor 1020 may load a command or data, which isreceived from at least one of other elements (e.g., the communicationmodule 1090), into a volatile memory 1032 to process the command or dataand may store the process result data into a nonvolatile memory 1034.

The memory 1030 may include, for example, the volatile memory 1032 orthe nonvolatile memory 1034. The volatile memory 1032 may include, forexample, a random access memory (RAM) (e.g., a dynamic RAM (DRAM), astatic RAM (SRAM), or a synchronous dynamic RAM (SDRAM)). Thenonvolatile memory 1034 may include, for example, a one-timeprogrammable read-only memory (OTPROM), a programmable read-only memory(PROM), an erasable programmable read-only memory (EPROM), anelectrically erasable programmable read-only memory (EEPROM), a maskROM, a flash ROM, a flash memory, a hard disk drive, or a solid-statedrive (SSD). In addition, the nonvolatile memory 1034 may be configuredin the form of an internal memory 1036 or the form of an external memory1038 which is available through connection only if necessary, accordingto the connection with the electronic device 1001. The external memory1038 may further include a flash drive such as compact flash (CF),secure digital (SD), micro secure digital (Micro-SD), mini securedigital (Mini-SD), extreme digital (xD), a multimedia card (MMC), or amemory stick. The external memory 1038 may be operatively or physicallyconnected with the electronic device 1001 in a wired manner (e.g., acable or a universal serial bus (USB)) or a wireless (e.g., Bluetooth)manner.

For example, the memory 1030 may store, for example, at least onedifferent software element, such as an instruction or data associatedwith the program 1040, of the electronic device 1001. The program 1040may include, for example, a kernel 1041, a library 1043, an applicationframework 1045 or an application program (interchangeably,“application”) 1047.

The input device 1050 may include a microphone, a mouse, or a keyboard.According to an embodiment, the keyboard may include a keyboardphysically connected or a keyboard virtually displayed through thedisplay 1060.

The display 1060 may include a display, a hologram device or aprojector, and a control circuit to control a relevant device. Thescreen may include, for example, a liquid crystal display (LCD), a lightemitting diode (LED) display, an organic LED (OLED) display, amicroelectromechanical systems (MEMS) display, or an electronic paperdisplay. According to an embodiment, the display may be flexibly,transparently, or wearably implemented. The display may include a touchcircuitry, which is able to detect a user's input such as a gestureinput, a proximity input, or a hovering input or a pressure sensor(interchangeably, a force sensor) which is able to measure the intensityof the pressure by the touch. The touch circuit or the pressure sensormay be implemented integrally with the display or may be implementedwith at least one sensor separately from the display. The hologramdevice may show a stereoscopic image in a space using interference oflight. The projector may project light onto a screen to display animage. The screen may be located inside or outside the electronic device1001.

The audio module 1070 may convert, for example, from a sound into anelectrical signal or from an electrical signal into the sound. Accordingto an embodiment, the audio module 1070 may acquire sound through theinput device 1050 (e.g., a microphone) or may output sound through anoutput device (not illustrated) (e.g., a speaker or a receiver) includedin the electronic device 1001, an external electronic device (e.g., theelectronic device 1002 (e.g., a wireless speaker or a wirelessheadphone)) or an electronic device 1006 (e.g., a wired speaker or awired headphone) connected with the electronic device 1001

The sensor module 1076 may measure or detect, for example, an internaloperating state (e.g., power or temperature) or an external environmentstate (e.g., an altitude, a humidity, or brightness) of the electronicdevice 1001 to generate an electrical signal or a data valuecorresponding to the information of the measured state or the detectedstate. The sensor module 1076 may include, for example, at least one ofa gesture sensor, a gyro sensor, a barometric pressure sensor, amagnetic sensor, an acceleration sensor, a grip sensor, a proximitysensor, a color sensor (e.g., a red, green, blue (RGB) sensor), aninfrared sensor, a biometric sensor (e.g., an iris sensor, a fingerprintsenor, a heartbeat rate monitoring (HRM) sensor, an e-nose sensor, anelectromyography (EMG) sensor, an electroencephalogram (EEG) sensor, anelectrocardiogram (ECG) sensor, a temperature sensor, a humidity sensor,an illuminance sensor, or an UV sensor. The sensor module 1076 mayfurther include a control circuit for controlling at least one or moresensors included therein. According to an embodiment, the sensor module1076 may be controlled by using the processor 1020 or a processor (e.g.,a sensor hub) separate from the processor 1020. In the case that theseparate processor (e.g., a sensor hub) is used, while the processor1020 is in a sleep state, the separate processor may operate withoutawakening the processor 1020 to control at least a portion of theoperation or the state of the sensor module 1076.

According to an embodiment, the interface 1077 may include a highdefinition multimedia interface (HDMI), a universal serial bus (USB), anoptical interface, a recommended standard 232 (RS-232), a D-subminiature(D-sub), a mobile high-definition link (MHL) interface, a SD card/MMCinterface, or an audio interface. A connector 1078 may physicallyconnect the electronic device 1001 and the electronic device 1006.According to an embodiment, the connector 1078 may include, for example,an USB connector, an SD card/MMC connector, or an audio connector (e.g.,a headphone connector).

The haptic module 1079 may convert an electrical signal into mechanicalstimulation (e.g., vibration or motion) or into electrical stimulation.For example, the haptic module 1079 may apply tactile or kinestheticstimulation to a user. The haptic module 1079 may include, for example,a motor, a piezoelectric element, or an electric stimulator.

The camera module 1080 may capture, for example, a still image and amoving picture. According to an embodiment, the camera module 1080 mayinclude at least one lens (e.g., a wide-angle lens and a telephoto lens,or a front lens and a rear lens), an image sensor, an image signalprocessor, or a flash (e.g., a light emitting diode or a xenon lamp).

The power management module 1088, which is to manage the power of theelectronic device 1001, may constitute at least a portion of a powermanagement integrated circuit (PMIC).

The battery 1089 may include a primary cell, a secondary cell, or a fuelcell and may be recharged by an external power source to supply power atleast one element of the electronic device 1001.

The communication module 1090 may establish a communication channelbetween the electronic device 1001 and an external device (e.g., thefirst external electronic device 1002, the second external electronicdevice 1004, or the server 1008). The communication module 1090 maysupport wired communication or wireless communication through theestablished communication channel. According to an embodiment, thecommunication module 1090 may include a wireless communication module1092 or a wired communication module 1094. The communication module 1090may communicate with the external device (e.g., the first externalelectronic device 1002, the second external electronic device 1004 orthe server 1008) through a first network 1098 (e.g. a wireless localarea network such as Bluetooth or infrared data association (IrDA)) or asecond network 1099 (e.g., a wireless wide area network such as acellular network) through a relevant module among the wirelesscommunication module 1092 or the wired communication module 1094.

The wireless communication module 1092 may support, for example,cellular communication, local wireless communication, and globalnavigation satellite system (GNSS) communication. The cellularcommunication may include, for example, long-term evolution (LTE), LTEAdvance (LTE-A), code division multiple access (CMA), wideband CDMA(WCDMA), universal mobile telecommunications system (UMTS), wirelessbroadband (WiBro), or global system for mobile communications (GSM). Thelocal wireless communication may include wireless fidelity (Wi-Fi),Wi-Fi Direct, light fidelity (Li-Fi), Bluetooth, Bluetooth low energy(BLE), Zigbee, near field communication (NFC), magnetic securetransmission (MST), radio frequency (RF), or a body area network (BAN).The GNSS may include at least one of a global positioning system (GPS),a global navigation satellite system (Glonass), Beidou NavigationSatellite System (Beidou), the European global satellite-basednavigation system (Galileo), or the like. In the disclosure, “GPS” and“GNSS” may be interchangeably used.

According to an embodiment, when the wireless communication module 1092supports cellar communication, the wireless communication module 1092may, for example, identify or authenticate the electronic device 1001within a communication network using the subscriber identificationmodule (e.g., a SIM card) 1096. According to an embodiment, the wirelesscommunication module 1092 may include a communication processor (CP)separate from the processor 2820 (e.g., an application processor (AP).In this case, the communication processor may perform at least a portionof functions associated with at least one of elements 1010 to 1096 ofthe electronic device 1001 in substitute for the processor 1020 when theprocessor 1020 is in an inactive (sleep) state, and together with theprocessor 1020 when the processor 1020 is in an active state. Accordingto an embodiment, the wireless communication module 1092 may include aplurality of communication modules, each supporting only a relevantcommunication scheme among cellular communication, short-range wirelesscommunication, or a GNSS communication scheme.

The wired communication module 1094 may include, for example, include alocal area network (LAN) service, a power line communication, or a plainold telephone service (POTS).

For example, the first network 1098 may employ, for example, Wi-Fidirect or Bluetooth for transmitting or receiving instructions or datathrough wireless direct connection between the electronic device 1001and the first external electronic device 1002. The second network 1099may include a telecommunication network (e.g., a computer network suchas a LAN or a WAN, the Internet or a telephone network) for transmittingor receiving instructions or data between the electronic device 1001 andthe second electronic device 1004.

According to embodiments, the instructions or the data may betransmitted or received between the electronic device 1001 and thesecond external electronic device 1004 through the server 1008 connectedwith the second network. Each of the external first and second externalelectronic devices 1002 and 1004 may be a device of which the type isdifferent from or the same as that of the electronic device 1001.According to various embodiments, all or a part of operations that theelectronic device 1001 will perform may be executed by another or aplurality of electronic devices (e.g., the electronic devices 1002 and1004 or the server 1008). According to an embodiment, in the case thatthe electronic device 1001 executes any function or serviceautomatically or in response to a request, the electronic device 1001may not perform the function or the service internally, but mayalternatively or additionally transmit requests for at least a part of afunction associated with the electronic device 1001 to any other device(e.g., the electronic device 1002 or 1004 or the server 1008). The otherelectronic device (e.g., the electronic device 1002 or 1004 or theserver 1008) may execute the requested function or additional functionand may transmit the execution result to the electronic device 1001. Theelectronic device 1001 may provide the requested function or serviceusing the received result or may additionally process the receivedresult to provide the requested function or service. To this end, forexample, cloud computing, distributed computing, or client-servercomputing may be used.

Various embodiments of the disclosure and terms used herein are notintended to limit the technologies described in the disclosure tospecific embodiments, and it should be understood that the embodimentsand the terms include modification, equivalent, and/or alternative onthe corresponding embodiments described herein. With regard todescription of drawings, similar elements may be marked by similarreference numerals. The terms of a singular form may include pluralforms unless otherwise specified. In the disclosure disclosed herein,the expressions “A or B”, “at least one of A and/or B”, “at least one ofA and/or B”, “A, B, or C”, or “at least one of A, B, and/or C”, and thelike used herein may include any and all combinations of one or more ofthe associated listed items. Expressions such as “first,” or “second,”and the like, may express their elements regardless of their priority orimportance and may be used to distinguish one element from anotherelement but is not limited to these components. When an (e.g., first)element is referred to as being “(operatively or communicatively)coupled with/to” or “connected to” another (e.g., second) element, itmay be directly coupled with/to or connected to the other element or anintervening element (e.g., a third element) may be present.

According to the situation, the expression “adapted to or configured to”used herein may be interchangeably used as, for example, the expression“suitable for”, “having the capacity to”, “changed to”, “made to”,“capable of” or “designed to”. The expression “a device configured to”may mean that the device is “capable of” operating together with anotherdevice or other components. For example, a “processor configured to (orset to) perform A, B, and C” may mean a dedicated processor (e.g., anembedded processor) for performing corresponding operations or ageneric-purpose processor (e.g., a central processing unit (CPU) or anapplication processor) which performs corresponding operations byexecuting one or more software programs which are stored in a memorydevice (e.g., the memory 1030).

The term “module” used herein may include a unit, which is implementedwith hardware, software, or firmware, and may be interchangeably usedwith the terms “logic”, “logical block”, “component”, “circuit”, or thelike. The “module” may be a minimum unit of an integrated component or apart thereof or may be a minimum unit for performing one or morefunctions or a part thereof The “module” may be implemented mechanicallyor electronically and may include, for example, an application-specificIC (ASIC) chip, a field-programmable gate array (FPGA), and aprogrammable-logic device for performing some operations, which areknown or will be developed.

According to various embodiments, at least a part of an apparatus (e.g.,modules or functions thereof) or a method (e.g., operations) may be, forexample, implemented by instructions stored in a computer-readablestorage media (e.g., the memory 1030) in the form of a program module.The instruction, when executed by a processor (e.g., a processor 1020),may cause the processor to perform a function corresponding to theinstruction. The computer-readable recording medium may include a harddisk, a floppy disk, a magnetic media (e.g., a magnetic tape), anoptical media (e.g., a compact disc read only memory (CD-ROM) and adigital versatile disc (DVD), a magneto-optical media (e.g., a flopticaldisk)), an embedded memory, and the like. The one or more instructionsmay contain a code made by a compiler or a code executable by aninterpreter.

Each element (e.g., a module or a program module) according to variousembodiments may be composed of single entity or a plurality of entities,a part of the above-described sub-elements may be omitted or may furtherinclude other elements. Alternatively or additionally, after beingintegrated in one entity, some elements (e.g., a module or a programmodule) may identically or similarly perform the function executed byeach corresponding element before integration. According to variousembodiments, operations executed by modules, program modules, or otherelements may be executed by a successive method, a parallel method, arepeated method, or a heuristic method, or at least one part ofoperations may be executed in different sequences or omitted.Alternatively, other operations may be added.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An electronic system, comprising: a user deviceincluding: a display, a microphone, a speaker, a communication circuit,a first memory, and a first processor electrically connected with thedisplay, the microphone, the speaker, the communication circuit, and thefirst memory; and a first server including: a communication interface, asecond memory, and a second processor electrically connected with thecommunication interface and the second memory, wherein the second memoryincludes an automatic speech recognition (ASR) module and a naturallanguage understanding (NLU) module, wherein the first memory storesinstructions that, when executed by the first processor, causes thefirst processor to: receive a voice input of a user through themicrophone, and transmit data associated with the voice input to thesecond processor through the communication circuit, wherein the secondmemory stores instructions that, when executed by the second processor,causes the second processor to: process the data by using at least oneof the ASR module or the NLU module, calculate at least one numericalvalue associated with at least one of the ASR module or the NLU moduleby using the data, and provide the at least one numerical value to thefirst processor, and wherein the first memory further storesinstructions that, when executed by the first processor, causes thefirst processor to provide information about the at least one numericalvalue on a user interface.
 2. An electronic device supporting a speechrecognition service, the electronic device comprising: a communicationmodule configured to communicate with at least one external device; amicrophone configured to receive a voice input according to user speech;a memory configured to store information about an operation of thespeech recognition service; a display configured to output a screenassociated with the operation of the speech recognition service; and aprocessor electrically connected with the communication module, themicrophone, the memory, and the display, wherein the processor isconfigured to: calculate a specified numerical value associated with theoperation of the speech recognition service, transmit information aboutthe numerical value to a first external device processing the voiceinput, and transmit a request for a function, which corresponds to thecalculated numerical value of at least one function associated with thespeech recognition service stepwisely provided from the first externaldevice depending on a numerical value, to the first external device torefine a function of the speech recognition service supported by theelectronic device.
 3. The electronic device of claim 2, wherein theprocessor is further configured to: assign a point to at least one of anautomatic speech recognition (ASR) module or a natural languageunderstanding (NLU) module included in the first external device inassociation with function execution of the first external device; andcalculate the numerical value based on collection of the assigned point.4. The electronic device of claim 3, wherein the processor is furtherconfigured to: collect at least one voice input information on whichspeaker-dependent speech recognition is performed by the ASR module;accumulate and calculate a user speech time corresponding to thecollected at least one voice input information; and assign the point tothe ASR module based on an accumulated amount of the user speech time.5. The electronic device of claim 3, wherein the processor is furtherconfigured to: assign the point to the ASR module based on generation ofa speaker-dependent recognition model with respect to wakeup-commandspeech associated with the operation of the speech recognition service.6. The electronic device of claim 3, wherein the processor is furtherconfigured to: in association with speech recognition execution of theASR module with respect to the voice input, if an error of the speechrecognition result is revised, assign the point to the ASR module basedon a speech recognition model update of the ASR module performed inresponse to the revision of the error.
 7. The electronic device of claim3, wherein the processor is further configured to: in association withderivation execution of user speech intent of the NLU module withrespect to the voice input, if user preference information provided froma user is applied to at least one of a domain, an intent, or a parameterassociated with the voice input obtained by the NLU module, assign thepoint to the NLU module based on application of the user preferenceinformation.
 8. The electronic device of claim 3, wherein the processoris further configured to: in association with derivation execution ofuser speech intent of the NLU module with respect to the voice input, ifat least one function response associated with function control of theelectronic device or function control of a second external deviceinteracting with the electronic device is set with respect to a specificuser speech intent to be derived by the NLU module, assign the point tothe NLU module based on the setting of the at least one functionresponse.
 9. The electronic device of claim 2, wherein the processor isfurther configured to: assign a point to a third external devicereceiving and storing at least one of information about the electronicdevice or information about a user of the electronic device from theelectronic device; and calculate the numerical value based on theassigned point.
 10. The electronic device of claim 9, wherein theprocessor is further configured to: receive and output query informationabout verification or check of the at least one information stored inthe third external device, from the first external device.
 11. Theelectronic device of claim 10, wherein the processor is furtherconfigured to: if the at least one information is verified or checked bya user feedback associated with the query information, assign the pointto the third external device based on the verification or the check ofthe at least one information.
 12. A speech recognition service operatingmethod of an electronic device, the method comprising: receiving a voiceinput according to user speech; calculating a specified numerical valuein association with an operation of the speech recognition service;transmitting at least one of information about the voice input orinformation about the numerical value to a first external deviceprocessing the voice input; transmitting a request for a function, whichcorresponds to the calculated numerical value of at least one functionassociated with the speech recognition service stepwisely provided fromthe first external device depending on a numerical value, to the firstexternal device; and receiving the function corresponding to thecalculated numerical value from the first external device to refine afunction of the speech recognition service.
 13. The method of claim 12,wherein the calculating includes: assigning a point to at least one ofan automatic speech recognition (ASR) module or a natural languageunderstanding (NLU) module included in the first external device inassociation with function execution of the first external device; andcalculating the numerical value based on collection of the assignedpoint.
 14. The method of claim 13, wherein the assigning includes:collecting at least one voice input information on whichspeaker-dependent speech recognition is performed by the ASR module;accumulating and calculating a user speech time corresponding to thecollected at least one voice input information; and assigning the pointto the ASR module based on an accumulated amount of the user speechtime.
 15. The method of claim 13, wherein the assigning includes:assigning the point to the ASR module based on generation of aspeaker-dependent recognition model with respect to wakeup-commandspeech associated with the operation of the speech recognition service.16. The method of claim 13, wherein the assigning includes: if an errorof a speech recognition result of the ASR module with respect to thevoice input is revised, assigning the point to the ASR module based on aspeech recognition model update of the ASR module performed in responseto the revision of the error.
 17. The method of claim 13, wherein theassigning includes: if user preference information provided from a useris applied to at least one of a domain, an intent, or a parameterassociated with the voice input obtained by the NLU module in anoperation of deriving user speech intent of the NLU module with respectto the voice input, assigning the point to the NLU module based onapplication of the user preference information.
 18. The method of claim13, wherein the assigning includes: if at least one function responseassociated with function control of the electronic device or functioncontrol of a second external device interacting with the electronicdevice is set with respect to a specific user speech intent to bederived by the NLU module deriving user speech intent associated withthe voice input, assigning the point to the NLU module based on thesetting of the at least one function response.
 19. The method of claim12, wherein the calculating includes: assigning a point to a thirdexternal device storing at least one of information about the electronicdevice or information about a user of the electronic device; andcalculating the numerical value based on the assigned point.
 20. Themethod of claim 19, wherein the assigning includes: receiving andoutputting query information about verification or check of at least oneinformation stored in the third external device, from the first externaldevice; verifying and checking the at least one information by a userfeedback associated with the query information; and assigning the pointto the third external device based on the verification or the check ofthe at least one information.